Systems and Methods for Haptically-Enabled Interactions with Objects

ABSTRACT

One illustrative computing device disclosed herein includes a sensor configured to detect a user interaction with a physical object and transmit a sensor signal associated with the user interaction. The illustrative computing device also includes a processor in communication with the sensor, the processor configured to: receive the sensor signal; determine a characteristic of the physical object based on the sensor signal; and determine a function based at least in part on the user interaction and the characteristic. The processor is also configured to determine a haptic effect associated with the function; and transmit a haptic signal associated with the haptic effect. The illustrative computing device further includes a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 15/490,253, filed Apr. 18, 2017 and entitled“Systems and Methods for Haptically-Enabled Interactions with Objects,”which is a continuation of U.S. patent application Ser. No. 14/577,565,filed Dec. 19, 2014 and entitled “Systems and Methods forHaptically-Enabled Interactions with Objections,” and is related to U.S.patent application Ser. No. 14/577,461, filed Dec. 19, 2014 and entitled“Systems and Methods for Object Manipulation with Haptic Feedback,” theentirety of each of which is hereby incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to the field of user interface devices.More specifically, the present invention relates to haptically-enabledinteractions with objects.

BACKGROUND

Humans are increasingly using computer-based systems to perform digitaltasks associated with products. For example, consumers may use mobiledevices (e.g., smartphones) to read reviews about a product, purchase aproduct, or add a product to a shopping list. It may be difficult andtime consuming, however, for a user to perform such digital tasks. Forexample, the user may need to interact with the small user interface(e.g., touchscreen display) of a mobile device to complete an onlinepurchase. Some digital tasks require multiple complicated steps that canbe frustrating for a user. Thus, there is a need for an improved userinterface that can allow users to perform such digital tasks quickly andeasily. It may be desirable to use haptic feedback (e.g., mechanicalvibrations) to improve such user interfaces.

SUMMARY

Embodiments of the present disclosure comprise haptically-enabledinteractions with objects. In one embodiment, a computing device of thepresent disclosure may comprise: a sensor configured to detect a userinteraction with a physical object and transmit a sensor signalassociated with the user interaction. The computing device may alsocomprise a processor in communication with the sensor, the processorconfigured to: receive the sensor signal; determine a characteristic ofthe physical object based on the sensor signal; and determine a functionbased at least in part on the user interaction and the characteristic.The processor may also be configured to determine a haptic effectassociated with the function; and transmit a haptic signal associatedwith the haptic effect. The computing device may further include ahaptic output device in communication with the processor, the hapticoutput device configured to receive the haptic signal and output thehaptic effect.

In another embodiment, a method of the present disclosure may comprise:receiving a sensor signal from a sensor, where the sensor signal isassociated with a user interaction with a physical object. The methodmay also comprise determining a characteristic of the physical objectbased on the sensor signal, and determining a function based at least inpart on the user interaction and the characteristic. The method mayfurther comprise determining a haptic effect associated with thefunction, and transmitting a haptic signal associated with the hapticeffect. A haptic output device may be configured to receive the hapticsignal and output the haptic effect. Yet another embodiment comprises acomputer-readable medium for implementing such a method.

These illustrative embodiments are mentioned not to limit or define thelimits of the present subject matter, but to provide examples to aidunderstanding thereof. Further embodiments are discussed in the DetailedDescription, and additional description is provided there. Advantagesoffered by various embodiments may be further understood by examiningthis specification and/or by practicing one or more embodiments of theclaimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure is set forth more particularly in theremainder of the specification. The specification makes reference to thefollowing appended figures.

FIG. 1 is a block diagram showing a system for haptically-enabledinteractions with objects according to one embodiment;

FIG. 2 is another block diagram showing a system for haptically-enabledinteractions with objects according to another embodiment;

FIG. 3 shows an embodiment of a system for haptically-enabledinteractions with objects;

FIG. 4 shows another embodiment of a system for haptically-enabledinteractions with objects;

FIG. 5A shows still another embodiment of a system forhaptically-enabled interactions with objects;

FIG. 5B shows yet another embodiment of a system for haptically-enabledinteractions with objects;

FIG. 6 shows another embodiment of a system for haptically-enabledinteractions with objects

FIG. 7 shows still another embodiment of a system for haptically-enabledinteractions with objects;

FIG. 8 shows yet another embodiment of a system for haptically-enabledinteractions with objects; and

FIG. 9 is a flow chart of steps for performing a method for providinghaptically-enabled interactions with objects according to anotherembodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to various and alternativeillustrative embodiments and to the accompanying drawings. Each exampleis provided by way of explanation, and not as a limitation. It will beapparent to those skilled in the art that modifications and variationscan be made. For instance, features illustrated or described as part ofone embodiment may be used in another embodiment to yield a stillfurther embodiment. Thus, it is intended that this disclosure includemodifications and variations that come within the scope of the appendedclaims and their equivalents.

Illustrative Examples of Haptically-Enabled Interactions with Objects

One illustrative embodiment of the present disclosure comprises acomputing device. The computing device comprises a processor coupled toa memory via a bus. In the illustrative embodiment, the computing deviceis configured to be worn by the user. For example, the computing devicemay comprise a ring configured to be worn on a user's finger.

In the illustrative embodiment, the computing device comprises a sensorconfigured to detect a user interaction with an object and transmitsensor signals to the processor. An object, as used herein, is anything(e.g., real or virtual) with which a user can potentially interact. Forexample, a user may pick up a 9 v battery from a drawer. The user maymake an “L” shaped gesture in the air whole holding the 9 v battery. Inthe illustrative embodiment, the computing device may detect the user'sinteraction with the 9 v battery. The computing device may determinethat the user interaction comprises an “L” shaped gesture, and that theobject comprises a 9 v battery. In the illustrative embodiment, thecomputing device determines, based at least in part on the userinteraction and the object, a function to perform. A function, as usedherein, comprises one or more digital operations. The computing devicemay then execute the function. For example, based on the “L” shapedgesture and the 9 v battery, the computing device may determine afunction comprising adding 9 v batteries to the user's shopping list.The computing device may then add 9 v batteries to the user's shoppinglist (e.g., on a shopping mobile application). In this manner, the usercan quickly and easily perform functions in the digital world byinteracting with objects in real space.

In the illustrative embodiment, the computing device further comprises ahaptic output device. The computing device may provide haptic feedbackto a user based on the user interaction, object, and/or function. Forexample, in the illustrative embodiment, the computing device outputs ahaptic effect comprising a short vibration to confirm the function hasbeen performed (e.g., upon 9 v batteries being added to the user'sshopping list). This may notify the user that the function has beenperformed.

The description of the illustrative embodiment above is provided merelyas an example. Various other embodiments of the present invention aredescribed herein and variations of such embodiments would be understoodby one of skill in the art. Advantages offered by various embodimentsmay be further understood by examining this specification and/or bypracticing one or more embodiments of the claimed subject matter.

Illustrative Systems for Haptically-Enabled Interactions with Objects

FIG. 1 is a block diagram showing a computing device 101 forhaptically-enabled interactions with objects according to oneembodiment. In some embodiments, the computing device 101 may comprise agraspable device (e.g., a smartphone, tablet, e-reader, and/or aportable gaming device). In other embodiments, the computing device 101may comprise a wearable device (e.g., a ring, a shoe, an armband, asleeve, a jacket, glasses, a glove, a watch, a wristband, a bracelet, anarticle of clothing, a hat, a headband, and/or jewelry).

In some embodiments, the components (e.g., the processor 202, network110, interaction sensor 132, sensor 130, etc.) of the computing device101 may be integrated into a single housing. In other embodiments, thecomponents may be distributed (e.g., among multiple housings orlocations) and in electrical communication with one another. Thecomputing device 101 may or may not comprise all of the componentsdepicted in FIG. 1. For example, in some embodiments, the computingdevice 101 may comprise the processor 102, bus 106, memory 104, network110, and haptic output device 118.

The computing device 101 comprises a processor 102 interfaced with otherhardware via bus 106. A memory 104, which can comprise any suitabletangible (and non-transitory) computer-readable medium such as RAM, ROM,EEPROM, or the like, may embody program components that configureoperation of the computing device 101. In some embodiments, thecomputing device 101 may further comprise one or more network interfacedevices 110, input/output (I/O) interface components 112, and additionalstorage 114.

Network interface device 110 can represent one or more of any componentsthat facilitate a network connection or otherwise facilitatecommunication between electronic devices. Examples include, but are notlimited to, wired interfaces such as Ethernet, USB, IEEE 1394, and/orwireless interfaces such as IEEE 802.11, Bluetooth, near-fieldcommunication (NFC) interfaces, RFID interfaces, or radio interfaces foraccessing cellular telephone networks (e.g., transceiver/antenna foraccessing a CDMA, GSM, UMTS, or other mobile communications network).

I/O components 112 may be used to facilitate connection to devices suchas one or more displays, touch sensitive surfaces 116, keyboards, mice,speakers, microphones, buttons, and/or other hardware used to input dataor output data. Storage 114 represents nonvolatile storage such asread-only memory, flash memory, ferroelectric RAM (F-RAM), magnetic,optical, or other storage media included in the computing device 101 orcoupled to processor 102.

The computing device 101 may comprise a touch sensitive surface 116.Touch sensitive surface 116 represents any surface that is configured tosense tactile input of a user. One or more touch sensors 108 areconfigured to detect a touch in a touch area (e.g., when an objectcontacts a touch sensitive surface 116) and transmit signals associatedwith the touch to processor 102. Any suitable number, type, orarrangement of touch sensors 108 can be used. For example, resistiveand/or capacitive sensors may be embedded in touch sensitive surface 116and used to determine the location of a touch and other information,such as pressure, speed, and/or direction. As another example, opticalsensors with a view of the touch sensitive surface 116 may be used todetermine the touch position. As still another example, the touchsensitive surface 116 may comprise a LED (Light Emitting Diode) fingerdetector mounted on the side of a display. In some embodiments, touchsensor 108 may be configured to detect multiple aspects of the userinteraction. For example, touch sensor 108 may detect both the speed andpressure of a user interaction, and incorporate this information intothe signal transmitted to the processor 102.

In some embodiments, the computing device 101 comprises a touch-enableddisplay that combines a touch sensitive surface 116 and a display of thedevice. The touch sensitive surface 116 may correspond to the displayexterior or one or more layers of material above components of thedisplay. In other embodiments, touch sensitive surface 116 may notcomprise (or otherwise correspond to) a display, depending on theparticular configuration of the computing device 101.

In some embodiments, the computing device 101 comprises an interactionsensor 132. The interaction sensor 132 is configured to detect aninteraction with the computing device 101 and/or an object (e.g., aproduct on a shelf in a store) by a user (e.g., using a finger, foot,hand, arm, head, leg, or other body part). In some embodiments, the userinteraction may comprise touching the object, gesturing in real space,gesturing using the object (e.g., picking up the object and moving it inreal space), or gesturing on an object (e.g., swiping a finger along asurface of the object). The interaction sensor 132 is further configuredto transmit a sensor signal associated with the interaction to processor102. The interaction sensor 132 may comprise an accelerometer,gyroscope, camera, radio frequency identification (RFID) tag or reader,indoor proximity system, NFC communication device, global positioningsystem (GPS) device, magnetometer, ultrasonic transducer, switch,button, optical sensor, light sensor, microphone, wireless interface(e.g., an IEEE 802.11 or Bluetooth interface), infrared sensor, depthsensor, and/or range sensor.

For example, in one embodiment, the interaction sensor 132 comprises awireless interface that is configured to detect the strength of awireless signal emitted by an object. The interaction sensor 132 maytransmit a sensor signal associated with the wireless signal strength tothe processor 102. Based on the wireless signal strength, the processor102 may determine, for example, whether the computing device 101 iswithin a predefined distance of the object. If so, the processor 102 maydetermine an interaction (e.g., coming within a predefined distance ofthe object) occurred.

In another embodiment, the interaction sensor 132 comprises a microphonepositioned to detect sounds associated with the manipulation of anobject. For instance, a user may shake a tin of coffee or a box ofcereal. The interaction sensor 132 may detect sounds associated with theshake. The interaction sensor 132 may transmit sensor signals associatedwith the sounds to the processor 102. Based on the sensor signals, theprocessor 102 may determine, for example, that the object has moved orthat a user interaction has otherwise occurred.

In still another embodiment, the interaction sensor 132 comprises acamera oriented toward an object. A user may make a gesture (e.g., acheck mark sign) in the air (e.g., with a body part, such as a finger,hand, arm, foot, head, or leg) near the object. The interaction sensor132 may capture images associated with the gesture and transmit sensorsignals to the processor 102. Based on the sensor signals, the processor102 may determine that a user interaction occurred. The processor 102may further analyze the sensor signals to determine the specific type ofgesture that occurred. For example, the processor 102 may analyze thesensor signals and determine that the user made a check mark in the airwith a finger.

In yet another embodiment, the interaction sensor 132 comprises anoptical sensor. An object may be positioned for blocking light fromreaching the interaction sensor 132. For example, the object may sit ontop of an interaction sensor 132 positioned in a shelf. Upon a usermoving the object or manipulating the object, the interaction sensor 132may detect a change in the amount of light. The interaction sensor 132may transmit sensor signals associated with the change in the amount oflight to the processor 102. Based on the sensor signals, the processor102 may determine that a user interaction occurred, or a characteristicof the user interaction (e.g., if the user moved the object a distancethat is above a threshold).

In some embodiments, the interaction sensor 132 is external to computingdevice 101 and in wired or wireless communication with the computingdevice 101. For example, the interaction sensor 132 may comprise acamera associated with a wearable device (e.g., glasses or a tie) and incommunication with the computing device 101. As another example, theinteraction sensor 132 may comprise a 3D imaging system (e.g., the 3Dimaging system commonly sold under the trademark Microsoft Kinect ®) ora LED-based tracking system positioned external to the computing device101 (e.g., on a shelf in a store) and in communication with thecomputing device 101.

The computing device 101 may further comprise one or more additionalsensor(s) 130. The sensor(s) 130 are configured to transmit sensorsignals to the processor 102. In some embodiments, the sensor 130 maycomprise, for example, a humidity sensor, ambient light sensor,gyroscope, GPS unit, accelerometer, range sensor, depth sensor,biosensor, camera, or temperature sensor. In some embodiments, thesensor 130 may be external to computing device 101 and in wired orwireless communication with the computing device 101. For example, thesensor 130 may comprise a biosensor coupled to a wearable device (e.g.,a ring or wristband). The biosensor may be configured to wirelesslytransmit sensor signals to the computing device 101, which may be, forexample, positioned in the user's pocket.

In some embodiments, the computing device 101 comprises a haptic outputdevice 118 in communication with processor 102. The haptic output device118 is configured to output a haptic effect in response to a hapticsignal. In some embodiments, the haptic output device 118 is configuredto output a haptic effect comprising a vibration, a change in aperceived coefficient of friction, a simulated texture, a change intemperature, a stroking sensation, an electro-tactile effect, or asurface deformation (e.g., a deformation of a surface associated withthe computing device 101). Further, some haptic effects may use multiplehaptic output devices 118 of the same or different types in sequenceand/or in concert. Although a single haptic output device 118 is shownin FIG. 1, embodiments may use multiple haptic output devices 118 of thesame or different type to produce haptic effects.

In some embodiments, the haptic output device 118 is external tocomputing device 101 and in communication with the computing device 101(e.g., via wired interfaces such as Ethernet, USB, IEEE 1394, and/orwireless interfaces such as IEEE 802.11, Bluetooth, or radiointerfaces). For example, the haptic output device 118 may be associatedwith (e.g., coupled to) a wearable device and configured to receivehaptic signals from the processor 102.

In some embodiments, the haptic output device 118 is configured tooutput a haptic effect comprising a vibration. The haptic output device118 may comprise, for example, one or more of a piezoelectric actuator,an electric motor, an electro-magnetic actuator, a voice coil, a shapememory alloy, an electro-active polymer, a solenoid, an eccentricrotating mass motor (ERM), or a linear resonant actuator (LRA).

In some embodiments, the haptic output device 118 is configured tooutput a haptic effect modulating the perceived coefficient of frictionof a surface associated with the haptic output device 118. In oneembodiment, the haptic output device 118 comprises an ultrasonicactuator. An ultrasonic actuator may vibrate at an ultrasonic frequency,for example 20 kHz, increasing or reducing the perceived coefficient ofan associated surface. In some embodiments, the ultrasonic actuator maycomprise a piezo-electric material.

In some embodiments, the haptic output device 118 uses electrostaticattraction, for example by use of an electrostatic actuator, to output ahaptic effect. The haptic effect may comprise a simulated texture, asimulated vibration, a stroking sensation, or a perceived change in acoefficient of friction on a surface associated with computing device101. In some embodiments, the electrostatic actuator may comprise aconducting layer and an insulating layer. The conducting layer may beany semiconductor or other conductive material, such as copper,aluminum, gold, or silver. The insulating layer may be glass, plastic,polymer, or any other insulating material. Furthermore, the processor102 may operate the electrostatic actuator by applying an electricsignal, for example an AC signal, to the conducting layer. In someembodiments, a high-voltage amplifier may generate the AC signal. Theelectric signal may generate a capacitive coupling between theconducting layer and an object (e.g., a user's finger, head, foot, arm,shoulder, leg, or other body part, or a stylus) near or touching thehaptic output device 118. Varying the levels of attraction between theobject and the conducting layer can vary the haptic effect perceived bya user.

In other embodiments, the haptic output device 118 may use reverseelectrovibration to generate a haptic effect. In such an embodiment, thehaptic output device 118 may include a conductor contacting a body partof the user. The processor 102 may operate haptic output device 118 byapplying an electric signal, for example an AC signal, to the user viathe conductor. The electrical signal can generate an electrical field onthe user's skin. If the user interacts with a physical object comprisinga conductor covered by an insulator, an electrical potential differencebetween the user's body (e.g., due to the electrical field generated bythe electrical signal) and the electrode may create an electrostaticattraction. Both the object and the user may be tied to a commonelectrical ground to generate the electrostatic attraction. In someembodiments, varying the properties of the electrical signal can causethe characteristics of the electrostatic attraction between the user andthe physical object to change. In some embodiments, this can vary thehaptic effect perceived by the user.

In some embodiments, the haptic output device 118 comprises adeformation device configured to output a deformation haptic effect. Thedeformation haptic effect may comprise raising or lowering portions of asurface associated with the computing device 101. For example, if thecomputing device 101 is positioned within a product (e.g., in a store),the deformation haptic effect may comprise raising portions of a surfaceof a product's packaging to generate a bumpy texture. In someembodiments, the deformation haptic effect may comprise bending,folding, rolling, twisting, squeezing, flexing, changing the shape of,or otherwise deforming a surface associated with the computing device101. For example, the deformation haptic effect may apply a force on thecomputing device 101 or a surface associated with the computing device101, causing it to bend, fold, roll, twist, squeeze, flex, change shape,or otherwise deform. For instance, if the computing device 101 ispositioned within a product's packaging, the deformation haptic effectmay comprise bending the top of a product's packaging toward a user.This may notify the user that a specific function has been performed(e.g., that the computing device 101 purchased the product via theInternet and the user is now free to take the product).

In some embodiments, the haptic output device 118 comprises fluidconfigured for outputting a deformation haptic effect (e.g., for bendingor deforming the computing device 101 or a surface associated with thecomputing device 101). For example, the fluid may comprise a smart gel.A smart gel comprises a fluid with mechanical or structural propertiesthat change in response to a stimulus or stimuli (e.g., an electricfield, a magnetic field, temperature, ultraviolet light, shaking, or apH variation). For instance, in response to a stimulus, a smart gel maychange in stiffness, volume, transparency, and/or color. In someembodiments, stiffness may comprise the resistance of a surfaceassociated with the computing device 101 against deformation. In someembodiments, one or more wires may be embedded in or coupled to thesmart gel. As current runs through the wires, heat is emitted, causingthe smart gel to expand or contract. This may cause the computing device101 or a surface associated with the computing device 101 to deform.

As another example, the fluid may comprise a rheological (e.g., amagneto-rheological or electro-rheological) fluid. A rheological fluidcomprises metal particles (e.g., iron particles) suspended in a fluid(e.g., oil or water). In response to an electric or magnetic field, theorder of the molecules in the fluid may realign, changing the overalldamping and/or viscosity of the fluid. This may cause the computingdevice 101 or a surface associated with the computing device 101 todeform.

In some embodiments, the haptic output device 118 comprises a mechanicaldeformation device. For example, in some embodiments, the haptic outputdevice 118 may comprise an actuator coupled to an arm that rotates adeformation component. The deformation component may comprise, forexample, an oval, starburst, or corrugated shape. The deformationcomponent may be configured to move a surface associated with thecomputing device 101 at some rotation angles but not others. Theactuator may comprise a piezo-electric actuator, rotating/linearactuator, solenoid, an electroactive polymer actuator, macro fibercomposite (MFC) actuator, shape memory alloy (SMA) actuator, and/orother actuator. As the actuator rotates the deformation component, thedeformation component may move the surface, causing it to deform. Insuch an embodiment, the deformation component may begin in a position inwhich the surface is flat. In response to receiving a signal fromprocessor 102, the actuator may rotate the deformation component.Rotating the deformation component may cause one or more portions of thesurface to raise or lower. The deformation component may, in someembodiments, remain in this rotated state until the processor 102signals the actuator to rotate the deformation component back to itsoriginal position.

Further, other techniques or methods can be used to deform a surfaceassociated with the computing device 101. For example, the haptic outputdevice 118 may comprise a flexible surface layer configured to deformits surface or vary its texture based upon contact from a surfacereconfigurable haptic substrate (including, but not limited to, e.g.,fibers, nanotubes, electroactive polymers, piezoelectric elements, orshape memory alloys). In some embodiments, the haptic output device 118is deformed, for example, with a deforming mechanism (e.g., a motorcoupled to wires), air or fluid pockets, local deformation of materials,resonant mechanical elements, piezoelectric materials,micro-electromechanical systems (“MEMS”) elements or pumps, thermalfluid pockets, variable porosity membranes, or laminar flow modulation.

In some embodiments, the haptic output device 118 is configured toremotely project haptic effects to a user. For example, the hapticoutput device 118 may comprise one or more jets configured to emitmaterials (e.g., solids, liquids, gasses, or plasmas) toward the user(e.g., toward the back of the user's hand). In one such embodiment, thehaptic output device 118 comprises a gas jet configured to emit puffs orstreams of oxygen, nitrogen, carbon dioxide, or carbon monoxide withvarying characteristics upon receipt of the haptic signal. As anotherexample, the haptic output device 118 may comprise one or moreultrasonic transducers or speakers configured to project pressure wavesin the direction of the user. In one such embodiment, upon the userinteracting with an object, the processor 102 may cause the hapticoutput device 118 to emit a concentrated pressure wave toward the user.The concentrated pressure wave may vibrate a portion of the user's body(e.g., the user's hand).

In some embodiments, the haptic output device 118 may be a portion ofthe housing of the computing device 101. In other embodiments, thehaptic output device 118 may be housed inside a flexible housingoverlaying a surface associated with the computing device 101 (e.g., thefront or back of the computing device 101). For example, the computingdevice 101 may comprise a watch. The haptic output device 118 maycomprise a layer of smart gel overlaying the interior of the band of thewatch. Upon actuating the haptic output device 118 (e.g., with anelectric current or an electric field), the smart gel may expand. Thismay cause the user to perceive a haptic effect comprising a squeezingsensation around the user's wrist.

Turning to memory 104, modules 124, 125, 126, and 128 are depicted toshow how a device can be configured in some embodiments to providehaptically-enabled interactions with objects. In this example, theobject interaction detection module 124 comprises code that configuresthe processor 102 to monitor the interaction sensor 132 and/or theadditional sensors 130 to determine if a user has interacted with anobject. The object interaction detection module 124 may comprise one ormore algorithms or lookup tables useable by the processor 102 todetermine whether a user is interacting with an object.

For example, the computing device 101 may be positioned within a productin a store. A user may, for example, lift the product off a shelf. Theobject interaction detection module 124 may comprise code that samplesthe sensor 130 (e.g., an accelerometer) to track the acceleration of theproduct. If the amount of acceleration exceeds a threshold, the objectinteraction detection module 124 may determine that the product is beingmanipulated (e.g., that the user has lifted the product).

As another example, the interaction sensor 132 may comprise a rangesensor oriented toward an object. The object interaction detectionmodule 124 may comprise code that configures the processor 102 toreceive data from the range sensor. The object interaction detectionmodule 124 may further comprise code that analyzes the data to determinewhether the user is within a certain distance of the object, which maybe indicative of a user interaction.

As still another example, the interaction sensor 132 may comprise a 3Dimaging system oriented toward an object. In one embodiment, the objectinteraction detection module 124 may comprise code for analyzing imagesfrom a 3D imaging system to determine whether a user is interacting withan object. Further examples of methods for detecting user interactionswith objects are described with respect to FIG. 9.

In some embodiments, the object may comprise a virtual object. Thevirtual object may be, for example, output on a touchscreen displaycomprising touch sensitive surface 116. The object interaction detectionmodule 124 may comprise code that configures the processor 102 to detecta user interaction with the virtual object. For instance, a user may tapon a location on the touch sensitive surface 116 associated with thevirtual object. The object interaction detection module 124 may receiveone or more sensor signals associated with the user interaction from thetouch sensor 108. In some embodiments, the sensor signals mayincorporate the location, pressure, direction, and/or speed of the userinteraction. The object interaction detection module 124 may determineone or more characteristics of the user interaction based on the sensorsignal.

As another example, the virtual object may be part of an augmentedreality environment output via, for instance, a touchscreen display,goggles, glasses, or contact lenses. The augmented reality environmentmay comprise camera data that has been supplemented (“augmented”) withvirtual content, such as text or images. For example, the interactionsensor 132 may comprise a camera. In some embodiments, the interactionsensor 132 may capture images of the user's living room, which theprocessor 102 may use to generate the augmented reality environment. Theprocessor 102 may further generate one or more virtual objects withinthe augmented reality environment. For example, the processor 102 mayinclude a virtual couch in the augmented reality environment. In someembodiments, a user may interact with the virtual object, for example,by touching or gesturing in an area in real space associated with thevirtual object. The object interaction detection module 124 maydetermine one or more characteristics of the user interaction, forexample, by analyzing images from the interaction sensor 132.

In some embodiments, the object interaction detection module 124comprises code that determines a characteristic (e.g., an amount ofpressure, speed, direction, location, type of gesture) associated withthe interaction. The object interaction detection module 124 maycomprise code that analyzes sensor signals from the sensor 130 and/orinteraction sensor 132 to determine the characteristic. For example, theobject interaction detection module 124 may comprise code for analyzingimages from a 3D imaging system to determine a type of gesture (e.g.,swipe, two-finger pinch, shake, etc.) made by the user.

In some embodiments, the object interaction detection module 124comprises code that determines a characteristic associated with theinteraction using swept frequency capacitive sensing. Swept frequencycapacitive sensing may comprise measuring the change in the capacitanceof an object (e.g., in voltage) at a plurality of frequencies as a userinteracts with the object. The object interaction detection module 124may generate a profile of the user interaction based on the changedcapacitances at the plurality of frequencies. In some embodiments, theobject interaction detection module 124 may compare the profile topredetermined interaction profiles. In some embodiments, eachpredetermined interaction profile may comprise a unique distribution ofchanges in capacitance at the plurality of frequencies and may beassociated with a different user interaction (e.g., a two finger pinch,a full hand grasp, or a single finger tap). For instance, apredetermined interaction profile associated with a two finger pinch maybe different than a predetermined interaction profile associated with afull hand grasp. In this manner, the object interaction detection module124 may be able to determine specifically how the user may beinteracting with the object.

In some embodiments, the object interaction detection module 124comprises code that determines a characteristic associated with theinteraction based on sound signals from the interaction sensor 132. Forexample, the interaction sensor 132 may comprise a microphone. Theobject interaction detection module 124 may receive sensor signals fromthe interaction sensor 132 and compare data from the sensor signals withone or more sound profiles. In some embodiments, each sound profile maycomprise sound data that is associated with a different userinteraction, for example, a user touching an object with a finger, aknuckle, a finger nail, or a palm. For instance, a sound profileassociated with a finger contacting an object may comprise a differentfrequency response than a sound profile associated with a knucklecontacting the object. In this manner, the object interaction detectionmodule 124 may be able to determine with which body part the user may beinteracting with the object.

In some embodiments, the object interaction detection module 124comprises code that determines a characteristic associated with theobject. The object may comprise a product in a store (e.g., a box ofcereal), a tool (e.g., a hammer or screw driver), a medical device(e.g., a trocar, needle, or heartbeat monitor), an automobile, a human,an animal, etc. The characteristic may comprise information about theobject. For example, depending on the type of object, the characteristicmay comprise a nutritional fact (if the object is a food product), size,weight, depth, color, texture, shape, dimension, price, discount,product rating (e.g., a grade or another comparative metric, such as a“star rating”), expiration date, function or utility (e.g., whatfunction the object performs or may be used for), available quantity orwhether the object is in stock, restriction on use, warranty, brand,manufacturer, producer, place of production, amount of power consumptionor other electrical characteristic, an amount of noise, a mechanicalcharacteristic, top speed, average speed, material (e.g., whether theobject is made of wood, glass, ceramic, plastic, and/or metal), and/orcompatibility information (e.g., what devices with which the object iscompatible) associated with the object.

In some embodiments, the object interaction detection module 124 maydetermine the characteristic based at least in part on informationprovided by the user or a third party. For example, a user may inputinto the computing device 101 data about one or more electronic devices(e.g., a TV and a video game system) that the user owns or desires. Insuch an embodiment, the object interaction detection module 124 mayconsult the data to determine if the object (e.g., a remote control) iscompatible with at least one of the electronic devices. Thus, in such anembodiment, the characteristic may comprise whether the object iscompatible with an electronic device owned or desired by the user.

In one embodiment, the object interaction detection module 124 maycomprise code that analyzes data from the interaction sensor 132 and/oradditional sensors 130 to determine the characteristic associated withthe object. For example, object interaction detection module 124 maycomprise code that receives images of the object from a camera andapplies Optical Character Recognition (OCR) to determine the contents oftext associated with the object. As another example, the objectinteraction detection module 124 may comprise code that analyzes imagesfrom the interaction sensor 132 to determine a height, width, color,size, shape, pattern, texture, name, type, QR code, barcode, label,logo, color scheme, shape, and/or another characteristic of the object.

In some embodiments, the object interaction detection module 124comprises code that analyzes data received via the network interfacedevice 110 to determine a characteristic of the object. For example, theobject may transmit a signal to the computing device 101 usingBluetooth, IEEE 802.11, RFID, or NFC. The signal may comprise acharacteristic of the object (e.g., the signal may comprise a name,type, brand, barcode, price, RFID code, or electrical characteristic).The object interaction detection module 124 may analyze parameters ofthe signal to determine the characteristic of the object.

In some embodiments, the object interaction detection module 124comprises code that determines a characteristic associated with theobject using the Internet or another network (e.g., a LAN). For example,the object interaction detection module 124 may comprise code forretrieving the name of a sports team (e.g., a baseball team) associatedwith an object (e.g., a baseball bat, helmet, or jersey) bycommunicating with one or more servers or webpages via the Internet. Asanother example, the object interaction detection module 124 maycomprise code for retrieving nutritional information associated with afood product in a store by communicating with a server associated withthe store (e.g., the store's local server).

The object interaction detection module 124 may comprise code thatidentifies the object based on the characteristic. For example, if theobject is a book, the object interaction detection module 124 maycomprise code that analyzes text associated with the book, a barcode,and/or a QR code to determine the name of the book. As another example,if the object is a car, the object interaction detection module 124 maycomprise code that analyzes images from a camera to determine the make,model, and/or year of the car.

Function determination module 125 comprises code that configures theprocessor 102 to determine a function to execute. The functiondetermination module 125 may comprise code that selects one or morefunctions to execute using one or more algorithms or lookup tables. Insome embodiments, a function may comprise purchasing an object (e.g.,via the Internet), opening a webpage, initiating printing of a document,opening a driving direction application, sending an e-mail or textmessage, determining information about the object (e.g., by querying oneor more servers), setting a virtual alarm clock, setting a parameter ofa navigation application (e.g., setting a starting point or destinationlocation), calling a phone number, copying text from a document, takinga picture, saving data, selecting a program option or setting, recordinga sound, adding an entry to a list, removing an entry from a list,outputting a sound, playing media content, opening an e-book, performinga calculation, sending data, and/or receiving data. The functiondetermination module 125 may also comprise code for executing thefunction.

In some embodiments, the function determination module 125 comprisescode for determining a function based on a characteristic of the userinteraction and a characteristic of the object. For example, thefunction determination module 125 may comprise a lookup table that mapsspecific user interactions (or general types of user interactions, suchas taps versus gestures) to specific functions. As another example, thefunction determination module 125 may comprise a lookup table that mapsspecific objects (or classes of objects) to specific functions.

In some embodiments, the function determination module 125 may comprisecode for communicating with another electronic device via a network todetermine a function. For example, in one such embodiment, the functiondetermination module 125 may comprise code to query one or more serversover the Internet to determine a function associated with a particularuser interaction and/or object.

In some embodiments, the object interaction detection module 124comprises code that determines the characteristics of a virtual object.For example, the object interaction detection module 124 may consultwith locations in memory 104 to determine characteristics of the virtualobject.

Haptic effect determination module 126 represents a program componentthat analyzes data to determine a haptic effect to generate. The hapticeffect determination module 126 may comprise code that selects one ormore haptic effects to output using one or more algorithms or lookuptables. In some embodiments, the haptic effect determination module 126comprises one or more algorithms or lookup tables useable by theprocessor 102 to determine a haptic effect.

In some embodiments, haptic effect determination module 126 comprisescode that determines a haptic effect to output based on a characteristicof the user interaction. For example, the haptic effect determinationmodule 126 may determine a haptic effect comprising a number of pulsedvibrations corresponding to the number of times a user rotated an object360 degrees. As another example, the haptic effect determination module126 may determine a haptic effect due to the user interaction comprisinga two finger pinch gesture along a surface of the object rather than atap on the object.

In some embodiments, haptic effect determination module 126 comprisescode that determines a haptic effect to output based on a characteristicof the object. For example, the haptic effect determination module 126may determine a haptic effect comprising a pulsed vibration if theobject is sharp. This may warn the user that the user is handling apotentially dangerous object. As another example, the haptic effectdetermination module 126 may comprise code that determines a hapticeffect comprising a wooden texture if the object comprises wood, a metaltexture if the object comprises metal, or a rubber texture if the objectcomprises rubber.

In some embodiments, haptic effect determination module 126 comprisescode that determines a haptic effect to output based on a characteristicof the function. For example, the haptic effect determination module 126may determine a haptic effect to be output prior to the function beingexecuted. This may alert the user that the function is going to beperformed or cannot be performed. As another example, the haptic effectdetermination module 126 may determine a haptic effect to be outputafter the function is executed. This may alert the user that thefunction was performed. As another example, the haptic effectdetermination module 126 may determine a haptic effect configured toindicate that an operation is ongoing or has failed (e.g., that thefunction has failed). For instance, if the function comprisesdownloading content (e.g., from a server), the haptic effectdetermination module 126 may determine a haptic effect comprising apulsed sensation indicative of the progress of the download. As yetanother example, the haptic effect determination module 126 maydetermine a different haptic effect if the function comprises purchasingan object using the Internet than if the function comprises adding anobject to a shopping list. This may alert the user to the function beingexecuted.

Haptic effect generation module 128 represents programming that causesprocessor 102 to generate and transmit haptic signals to the hapticoutput device 118 to generate the selected haptic effect. For example,the haptic effect generation module 128 may access stored waveforms orcommands to send to the haptic output device 118 to create the desiredeffect. In some embodiments, the haptic effect generation module 128 maycomprise algorithms to determine the haptic signal. Further, in someembodiments, haptic effect generation module 128 may comprise algorithmsto determine target coordinates for the haptic effect (e.g., coordinatesfor a location on the computing device 101 or object at which to outputa haptic effect).

Although the modules 124, 125, 126, 128 are depicted in FIG. 1 asprogram components within the memory 104, in some embodiments, themodules 124, 125, 126, 128 may comprise hardware. For example, modules124, 125, 126, 128 may comprise analog to digital converters,processors, microcontrollers, comparators, amplifiers, transistors, andother analog or digital circuitry.

FIG. 2 is another block diagram showing a system for haptically-enabledinteractions with objects according to another embodiment. The system200 may comprise one or more remote haptic output devices 206. Thesystem 200 may also comprise one or more computing devices 202 a-g.

In the embodiment shown in FIG. 2, the system 200 comprises a wearablecomputing device 202 a, a packaging computing device 202 b, a shelfcomputing device 202 c, a store computing device 202 d, a floorcomputing device 202 e, a carrier computing device 202 f, and agraspable computing device 202 g. A wearable computing device 202 a maycomprise a computing device associated with a wearable device (e.g., aring) that is configured to be worn by the user. A packaging computingdevice 202 b may comprise a computing device that is at least partiallyembedded within an object's packaging (e.g., an object's wrapper orcontainer). A shelf computing device 202 c may comprise a computingdevice that is at least partially embedded within or coupled to a shelf(e.g., for holding a product for sale in a store or a bookshelf). Astore computing device 202 d may comprise a computing device associatedwith a seller, manufacturer, and/or distributer of an object. A floorcomputing device 202 e may comprise a computing device that is at leastpartially embedded within or coupled to a floor (e.g., within one ormore floor tiles of a store). A carrier computing device 202 f maycomprise a computing device that is at least partially embedded withinor coupled to a carrying device for holding an object (e.g., a shoppingcart, basket, bag, or backpack). A graspable computing device 202 g maycomprise a computing device associated with a graspable device (e.g., amobile phone) configured to be grasped by the user. These computingdevices 202 a-g are described in greater detail below.

The computing devices 202 a-g and remote haptic output device(s) 206 maybe connected to a network 204. The network 204 may be any suitablenumber or type of networks or links, including, but not limited to, adial-up network, a local area network (LAN), wide area network (WAN),public switched telephone network (PSTN), a cellular network, a WiFinetwork, the Internet, an intranet or any combination of hard-wiredand/or wireless communication links. In some embodiments, the network204 is a single network. In other embodiments, the network 204 maycomprise two or more networks.

The computing devices 202 a-g and remote haptic output device(s) 206 maydirectly communicate with each other and/or may communicate with eachother via the network 204. For example, the shelf computing device 202 cmay directly wirelessly communicate with store computing device 202 d(e.g., using Bluetooth). Further, the computing devices 202 a-g andremote haptic output device(s) 206 may communicate with one or moreremote servers (e.g., cloud servers, webservers, or other servers),databases, and/or computing devices via the network.

As described above, the system 200 may comprise a wearable computingdevice 202 a. The wearable computing device 202 a may be associated witha wearable device, comprising, for example, a watch, wristband, hat,sleeve, jacket, collar, glasses, glove, ring, articles of clothing,headband, and/or jewelry. Upon a user interacting with an object, thewearable computing device 202 a may execute a function and/or output ahaptic effect to a body part of the user (e.g., the user's wrist, arm,leg, foot, hand, finger, neck, head, or chest).

For example, the wearable computing device 202 a may comprise a watch.Upon the user tapping the watch with a book (e.g., that the user wantsto read), the wearable computing device 202 a may detect the tap andpurchase an electronic version of the book (e.g., from the storecomputing device 202 d) via the network 204. The wearable computingdevice 202 a may output a haptic effect configured to notify the userthat the book has been purchased. The wearable computing device 202 amay also communicate with another electronic device (not shown), such asthe user's e-reader, and initiate the download of the electronic book tothe electronic device. As the book downloads to the electronic device,the wearable computing device 202 a may output haptic effects associatedwith the progress of the download. For example, the wearable computingdevice 202 a may output a vibration with an increasing frequency as thedownload progresses. Once the book has finished downloading, thewearable computing device 202 a may output another haptic effect, suchas a jolt sensation. This may notify the user that the book has beendownloaded to the electronic device.

As described above, the system 200 may comprise a packaging computingdevice 202 b. For example, the packaging computing device 202 b may beembedded within or coupled to the packaging of an object. The packagingcomputing device 202 b may comprise a sensor (e.g., an accelerometer,pressure sensor, capacitive sensor, resistive sensor, 3D imaging system,or a LED-based tracking system) configured to detect a user interactionwith an object and/or the object's packaging. For example, the sensormay comprise a pressure sensor embedded within a portion of the object'spackaging. The pressure sensor may be able to detect contact by a user.Based on signals from the sensor, the packaging computing device 202 bmay be configured to execute a function and/or output a haptic effect(e.g., directly to the user's hand or finger as the user contacts theobject).

For example, the object may comprise a drug from a pharmacy. Thepackaging computing device 202 b may be, for example, embedded withinthe cap of the drug container. Upon the user picking up the drugcontainer and rotating it 360 degrees, the packaging computing device202 b may detect the user interaction and order a refill of the drug viathe network 204. In some embodiments, if the refill is successfullyordered, the packaging computing device 202 b output a haptic effectcomprising three pulses. If the refill cannot be ordered (e.g., becausethe user has exceeded a refill quota), the packaging computing device202 b may output a jolt sensation or no haptic effect.

As described above, the system 200 may comprise a shelf computing device202 c. The shelf computing device 202 c may comprise a sensor (e.g., apressure sensor, light or optical sensor, button, switch, accelerometer,position sensor, 3D imaging system, or a LED-based tracking system)configured to detect a user interaction with an object. For example, thesensor may comprise a pressure sensor positioned under an object on theshelf. The sensor may be configured to detect a user lifting an objectoff the shelf. Based on signals from the sensor, the shelf computingdevice 202 c may be configured to execute a function and/or output ahaptic effect. For example, a user may take an object off a shelf. Theshelf computing device 202 c may determine that the user took the lastof the particular object available from the shelf. The shelf computingdevice 202 c may notify (e.g., via text message, e-mail, or other alertsystem) a store manager that the shelf needs to be restocked.

As described above, the system 200 may comprise a store computing device202 d. The store computing device 202 d may be associated with a seller,manufacturer, and/or distributer of an object. For example, the storecomputing device 202 d may be owned or operated by the store in whichthe object is for sale. The store computing device 202 d may facilitatepurchasing the object (e.g., process credit card transactions, coupons,discounts, store rewards cards, and/or otherwise conduct a purchase). Insome embodiments, the store computing device 202 d may comprise data(e.g., in a database) associated with the object. For example, the storecomputing device 202 d may comprise a name, weight, size, price,discount, manufacturer, list of ingredients, and/or other characteristicassociated with the object. One or more other computing devices 202 a-c,202 e-f may communicate with the store computing device 202 d todetermine a characteristic associated with the object. For example, thewearable computing device 202 a may transmit a query associated with theobject to the store computing device 202 d. The store computing device202 d may receive the query and consult a database to determine thequantity of the object available for sale in the store. The storecomputing device 202 d may transmit a signal associated with thequantity to the wearable computing device 202 a.

In some embodiments, the store computing device 202 d may be configuredto detect a user interaction with an object. For example, the storecomputing device 202 d may be in wired or wireless communication with aninteraction sensor. The interaction sensor may comprise, for example, adepth sensor oriented toward an object. Upon the user interacting withan object, the store computing device 202 d may determine a function toexecute (e.g., send the user an e-mail with a discount coupon for theobject). Based on the object and/or function, the store computing device202 d may cause the remote haptic output device 206 and/or anothercomputing device 202 a-c, 202 e-f to output a haptic effect. Forexample, the store computing device 202 d may transmit a signal to awearable computing device 202 a configured to cause the wearablecomputing device 202 a to output a short vibration. This may alert theuser that the user is going to receive a discount coupon for the object.

As described above, the system 200 may comprise a floor computing device202 e. The floor computing device 202 e may execute a function and/oroutput a haptic effect (e.g., a vibration) upon the user interactingwith an object. For example, upon a user touching a product in a store,the floor computing device 202 e may determine if the product is onsale. If so, the floor computing device 202 e may output ahigh-magnitude vibration. The user may perceive the haptic effect viathe user's feet or another body part. In other embodiments, the floorcomputing device 202 e may comprise a computing device in communicationwith a remote haptic output device 206 that is embedded in or coupled tothe floor. Upon the user interacting with an object, the floor computingdevice 202 e may cause the remote haptic output device 206 to output ahaptic effect (e.g., to the user's feet or another body part).

As described above, the system 200 may comprise a carrier computingdevice 202 f. In some embodiments, the carrier computing device 202 fexecutes a function and/or outputs a haptic effect upon a userinteracting with an object. For example, a user may be holding ashopping basket comprising a carrier computing device 202 f. The carriercomputing device 202 f may comprise one or more sensors configured todetect if a user has placed product in the shopping cart. Upon the userplacing a product in the shopping cart, the carrier computing device 202f may detect the product and remove the product from the user's digitalshopping list. The carrier computing device 202 f may also determine theprice of the product (e.g., by communicating with store computing device202 d or via the Internet). The carrier computing device 202 f mayupdate a total price associated with the objects in the shopping cart.In some embodiments, if the total price exceeds a threshold (e.g.,$150), the carrier computing device 202 f may output a haptic effect.This may notify the user that the user has exceeded a spending limit(e.g., input or customizable by the user).

As described above, the system 200 may comprise a hand-held or graspablecomputing device 202 g. The graspable computing device 202 g maycomprise, for example, a mobile device, e-reader, tablet, wand, stylus,or pen. In some embodiments, the graspable computing device 202 g maybe, for example, positioned in the user's pants pocket (or shirtpocket). Upon user interacting with an object, the graspable computingdevice 202 g may execute a function and/or output a haptic effect to theuser's thigh (or chest). In other embodiments, the user may interactwith the object using the graspable computing device 202 g as anintermediary, for example, as described below with respect to FIG. 5B.

The system 200 may additionally or alternatively comprise one or morecomputing devices positioned in other locations or configurations. Forexample, the system 200 may comprise a computing device positionedwithin the object. For instance, the object itself may comprise acomputing device. In some embodiments, upon a user interacting with theobject (e.g., picking up the object), the computing device may execute afunction and/or output a haptic effect (e.g., to the user's hand).

In some embodiments, any of the computing devices 202 a-g may cause anyof the other computing device 202 a-g (and/or the remote haptic outputdevice 206) to output a haptic effect. For example, a user may beshopping for a toy in a toy store. A packaging computing device 202 bmay detect the user tapping on the box of a toy. For instance, thepackaging computing device 202 b may comprise a pressure sensorconfigured to detect a user interaction. Upon the user tapping on thebox of the toy, the packaging computing device 202 b may detect the userinteraction based on sensor signals from the pressure sensor. Thepackaging computing device 202 b may transmit a signal to a wearablecomputing device 202 a, e.g., worn on the user's wrist. The signal maycause the wearable computing device 202 a to check the user's bankbalance. In some embodiments, if the user has enough money to purchasethe toy, the wearable computing device 202 a may output a vibration.

As another example, the packaging computing device 202 b may detect theuser picking an object (e.g., a box of cereal) up off of a shelf. Forinstance, the packaging computing device 202 b may comprise anaccelerometer. Upon the user picking up the object, the packagingcomputing device 202 b may detect the user interaction based on sensorsignals from the accelerometer. The packaging computing device 202 b maytransmit a signal to the remote haptic output device 206. The signal maybe configured to cause the remote haptic output device 206 to output apuff of air at the user's head. This may allow the packaging computingdevice 202 b to cause a haptic effect to be output to a body part of theuser (e.g., the user's head) to which the packaging computing device 202b may otherwise be unable to output haptic effects.

As still another example, the user may be wearing a computing device 202a comprising a camera. Upon the wearable computing device 202 adetecting that a user is holding an egg (e.g., via the camera), thewearable computing device 202 a may determine whether the egg is thelast egg in the user's refrigerator (e.g., by communicating with therefrigerator, or analyzing images of the refrigerator and/or eggcarton). If so, the wearable computing device 202 a may output a hapticeffect comprising a short vibration. The user may perceive the shortvibration as a query as to whether the user would like to purchase moreeggs. The user may rotate the egg by 90 degrees, which may indicate thatthe user would like to purchase additional eggs. The wearable computingdevice 202 a may detect the rotation and transmit a signal to a storecomputing device 202 d (e.g., at a remote grocery store). The signal maycause the store computing device 202 d to process the purchase andarrange for the eggs to be delivered to the user's home. Upon completionof the order, the store computing device 202 d may transmit aconfirmation signal to the wearable computing device 202 a. The signalmay cause the wearable computing device 202 a to determine a hapticeffect such as a pulsed vibration. The wearable computing device 202 amay then output this haptic effect via a haptic output device. Thehaptic effect may indicate that the purchase was successful.

FIG. 3 shows an embodiment of a system for haptically-enabledinteractions with objects. The system 300 comprises an object 306positioned on a shelf 308. The object 306 may comprise a toy, computer,mobile device, automotive component, movie, video game, video gameconsole, appliance, television, medical device, mechanical or electricalcomponent, remote control, food, etc. The system 300 may be associatedwith a store, warehouse, restaurant, medical center, garage, house,office, apartment, or other location.

In the embodiment shown in FIG. 3, an interaction sensor 315 (e.g., a 3Dimaging system) is oriented toward the object 306. The interactionsensor 315 may detect a user interaction (e.g., tap, touch, gesture on,shake, lift, gesturing toward, etc.) with the object 306. The user mayinteract with the object 306, for example, to perform a function (e.g.,open a webpage, purchase the object 306, receive a discount associatedwith the object 404, add the object 306 to a list, remove the object 306from a list, download a file associated with the object 306, etc.)associated with the object 306. The interaction sensor 315 may transmita sensor signal to one or more computing devices 304, 310, 312, 318.Based on the user interaction, one or more of the computing devices 304,310, 312, 318 may execute the function. The one or more computingdevices 304, 310, 312, 318 may then output a haptic effect, e.g.,associated with the function.

In this example, the system 300 comprises a wearable computing device304 that includes a wristband or watch. The system 300 also comprises apackaging computing device 310, a shelf computing device 312, a floorcomputing device 318, and a remote haptic output device 314. The remotehaptic output device 314 may be in wired or in wireless communicationwith the computing devices 304, 310, 312, 318. In some embodiments, uponthe user interacting with the object 306, one or more computing devices304, 310, 312, 318 may execute a function associated with theinteraction and/or object 306. For example, the function may comprisecommunicating with a remote server to determine whether the object 306is on sale. Further, one or more of the computing devices 304, 310, 312,318 may cause the remote haptic output device 314 to output a remotehaptic effect. For example, if the object 306 is on sale, the wearablecomputing device 304 may transmit a signal configured to cause theremote haptic output device 314 to output an ultrasonic pressure wavedirected toward the user's hand. The user may perceive the ultrasonicpressure wave as a vibration on the user's hand. The haptic effect mayindicate to the user that the object 306 is on sale. In someembodiments, the wearable computing device 304 may not cause the remotehaptic output device 314 to output a haptic effect if the object 306 isnot on sale. The lack of a haptic effect may indicate to the user thatthe object 306 is not on sale.

FIG. 4 shows another embodiment of a system for haptically-enabledinteractions with objects. In this example, the user 402 is wearing acomputing device 406 comprising a ring. The user 402 is interacting withan object 404 by making a gesture in the air. In some embodiments, thegesture may be oriented toward the object 404 or within a predefineddistance (e.g., 2 feet) from the object 404. For example, the user 402may wave at, point at, reach for, and/or use a finger to draw a symbolin the air in front of the object 404. The computing device 406 maydetect the user interaction and determine the user's gesture. Based onthe characteristics of the gesture and/or the object 404, the computingdevice 406 may determine a function to execute. The computing device 406may then execute the function and/or output a haptic effect associatedwith the function.

For example, the user 402 may make a gesture in the air to execute afunction associated with the object 404, e.g., to add an address 408(e.g., a return address) that is printed on the object 404 to a contactlist or address book. In the embodiment shown in FIG. 4, the user 402 isdrawing a “C” shape in the air in front of an object 404. The computingdevice 406 may detect a gesture and determine that the user 402 ismaking a “C” shaped gesture. The computing device 406 associate the “C”shaped gesture with adding a contact to the user's contact list oraddress book. The computing device 406 may further determine that theobject 404 comprises a piece of mail. Based on the “C” gesture, thecomputing device 406 may determine an address (e.g., an address 408)associated with the piece of mail (e.g., written on the piece of mail).The computing device 406 may add the address 408 to the user's contactlist or address book. In some embodiments, the computing device 406 mayoutput a haptic effect comprising a light vibration, e.g., to indicatethat the address was successfully added to the user's contact list.

In some embodiments, the object 404 may comprise a business card. Theaddress 408 may comprise a phone number (e.g., associated with thebusiness). The user 402 may make a gesture in the air (e.g., a “C”shape) to execute a function associated with the object 404, e.g., tocall the phone number. In some embodiments, the computing device 406 maydetect the gesture and associate the gesture with calling a phonenumber. The computing device 406 may determine a phone number associatedwith the address 408 and call the phone number. In some embodiments, thecomputing device 406 may further output a haptic effect, e.g., toindicate that the computing device 406 has initiated a phone call.

In some embodiments, the computing device 406 may be in a “wait mode.”In the wait mode, the computing device 406 may wait to detect a specificuser interaction with an object 404 (e.g., tapping on the object 404,squeezing the object 404 twice, or making a pinching gesture orientedtoward the object 404) before executing any functions. For example, ifthe computing device 406 is in the wait mode and detects a userinteraction comprising a “C” shaped gesture, the computing device 406may not add an address to the user's contact list. In some embodiments,upon detecting the specific user interaction, the computing device 406may enter a “function execution” mode. Once in the function executionmode, the computing device 406 may execute functions associated withsubsequent user interactions with the object 404. For example, uponentering the function execution mode, if the computing device 406detects a “C” shape gesture by the user 402 oriented toward the object404, the computing device 406 may add an address 408 associated with theobject 404 to the user's contact list. In some embodiments, thecomputing device 406 may return to the wait mode, for example, afterdetecting a specific user interaction (e.g., the user 402 shaking ahand) or after a period of time (e.g., 10 seconds). In some embodiments,this may allow the user 402 to interact with objects 404 withouttriggering functions until the user intends to trigger such functions.

FIG. 5A shows still another embodiment of a system for objectmanipulation with haptic feedback. In this example, the user 502 iswearing a watch comprising the computing device 506. The user 502 isinteracting with an object 504 (e.g., a milk carton) by contacting(e.g., tapping or touching) the object 504. The computing device 506 maydetect the contact via interaction sensor 510 (e.g., embedded within thecomputing device 506). The interaction sensor 510 may comprise, forexample, a camera oriented toward the object 504. Based on thecharacteristics of the contact and/or the object 504, the computingdevice 506 may determine a function associated with the object 404. Thecomputing device 506 may then execute the function and/or output ahaptic effect associated with the function via a haptic output device512.

In some embodiments, the user may use an intermediary object (e.g., astylus, pen, cane, or wand) for an interaction, and the computing device506 may detect such an interaction. For example, as shown in FIG. 5B,the user 502 may contact the object 504 with an intermediary object 514comprising a mobile phone. The computing device 506 may detect thecontact and execute an associated function. For example, the computingdevice 506 may detect the contact and call a phone number associatedwith the object 504 (e.g., the manufacturer's phone number, a supportphone number, or a product information phone number). In someembodiments, the intermediary object 514 comprises the computing device506 (e.g., rather than the computing device 506 being separate from theintermediary object 514). For example, rather than the user 502 wearingthe computing device 506 on a wrist, the mobile phone may be thecomputing device.

The computing device 506 may detect a contact anywhere on the object504, or a contact with a specific location (e.g., the label) on theobject 504. The computing device 506 may detect a location of thecontact. For example, the computing device 506 may detect which portionof the object 504 (e.g., the top, bottom, left side, right side, front,back, a label, an image, a logo, a piece of text, etc.) was contacted bythe user. In some embodiments, the computing device 506 may execute afunction upon the user interacting with a specific portion of the object504. For example, the user 502 may contact a manufacturer's logo on theobject 504. The computing device 506 may detect the interaction with themanufacturer's logo and open a browser to the manufacturer's website.Upon the user interacting with a different portion of the object 504(e.g., a nutritional label), the computing device 506 may execute adifferent function (e.g., the computing device 506 may determine thefood ingredients making up the object 504) or may not execute a functionat all.

In some embodiments, the computing device 506 may detect the contact anddetermine information associated with the object 504. For example, theuser 502 may contact an object 504 to receive information associatedwith the price of the object 504. The computing device 506 may detectthe contact and determine the price of the object 504 (e.g., by queryingone or more servers over a network). In some embodiments, if the priceis below a threshold (e.g., input by the user), the computing device 506may output a haptic effect (e.g., a pulsed vibration). This may indicateto the user 502 that the object 504 has a price that is acceptable tothe user 502. In other embodiments, the computing device 506 may performa price comparison with other local stores. The computing device 506 mayoutput a haptic effect (e.g., comprising two pulsed vibrations)configured to notify the user 502 that the price may be cheaper atanother local store (e.g., a store within a 15 mile radius) or throughan online retailer.

In some embodiments, the computing device 506 may detect multiple userinteractions (or a single user interaction comprising multiple contacts)with the object 504. For example, in some embodiments, the object 504may comprise a map. The user 502 may touch the map at a starting pointon the map with a finger. The user 502 may sequentially orsimultaneously (with another finger) touch the map at a destinationlocation. The computing device 506 may detect the user interactions andexecute one or more functions. For example, the computing device 506 maydetect the two touches and retrieve driving directions from the startingpoint to the destination location. As another example, the computingdevice 506 may detect a user interaction with a business card (oranother object 504) comprising an address. For instance, the computingdevice 506 may detect the user 502 tapping on the business card twotimes. The computing device 506 may detect the two taps and, forexample, set the destination of a navigation application to the address.

In some embodiments, the object 504 may comprise a virtual object. Thevirtual object may be, for example, output on the display 508. In onesuch embodiment, the virtual object comprises images representing fooditems in the user's fridge. For instance, the virtual object maycomprise an image of a bottle of milk, carrots, a piece of meat, etc.The computing device 506 may be configured to detect a user interactionwith the virtual object. For example, a user 502 may double tap on thedisplay (e.g., which may be a touchscreen display) on a locationassociated with the bottle of milk. Based on the user interaction, thecomputing device 506 may determine information associated with thevirtual object. For example, based on the double tap, the computingdevice 506 may determine, for example, how many bottles of milk the user502 has in the user's refrigerator. The computing device 506 may outputa haptic effect associated with the information. For example, thecomputing device 506 may output a number of pulsed vibrations equal tothe number of bottles of milk in the user's refrigerator.

In some embodiments, the function comprises outputting a sound. Forexample, the computing device 506 may detect a user interaction with anobject 504 and determine a commercial associated with the object 504. Insome embodiments, the computing device 506 may output audio associatedwith the commercial (e.g., the audio track associated with thecommercial). As another example, the computing device 506 may detect auser interaction with an object 504 (e.g., a CD or tape) and determine asong associated with the object 504. In some embodiments, the computingdevice 506 may output the song. The computing device 506 may also outputa haptic effect comprising pulsed vibrations that are timed withspecific portions of the song.

As another example, the computing device 506 may detect a userinteraction with an object 504 and determine the price of the object504. In one embodiment, if the price is below a threshold (e.g., inputby the user), the computing device 506 may output the sound of a cashregister opening (e.g., “ching ching”). In some embodiments, thecomputing device 506 may also output a haptic effect comprising a pulsedvibration. The sound and/or haptic effect may notify the user that theobject 504 has an acceptable price.

In some embodiments, the function comprises outputting data on a display508. For example, the computing device 506 may play a video on thedisplay 508 in response to a user interaction with an object 504. Forinstance, the user may interact with an object comprising a movieposter. In some embodiments, the computing device 506 may play a videoof a movie trailer associated with the movie poster. In someembodiments, the computing device 506 may also output one or more hapticeffects, e.g., timed with specific portions of the movie trailer (e.g.,the computing device 506 may output an intense vibration during a carchase in the movie trailer).

As another example, a user 502 may tap on an object 504 in a store. Thecomputing device 506 may determine the quantity of the object 504 thestore has left in stock (e.g., by querying a server associated with thestore). In some embodiments, the computing device 506 may output thequantity on a display 508. The computing device 506 may also output ahaptic effect associated with the quantity (e.g., if the quantity isbelow a threshold).

As still another example, a user 502 may tap on an object 504 in astore. The computing device 506 may determine the price of the object504 in the store and the average price of the object 504 (e.g., asdetermined based on prices from a plurality of stores). If the price inthe store is lower than the average price of the object 504, thecomputing device 506 may output a single dollar sign on the display 508.In some embodiments, the computing device 506 may also output a hapticeffect comprising a stroking sensation. If the price is higher than theaverage price of the object 504, the computing device 506 may outputthree dollar signs on the display 508. In some embodiments, thecomputing device 506 may also output a haptic effect comprising astinging sensation. If the user 502 makes another gesture (e.g., if theuser 502 taps the object 504 again), the computing device 506 maypurchase the object 504. In some embodiments, upon completion of thepurchase, the computing device 506 may output a notification on thedisplay 508. The notification may alert the user that the object 504 hasbeen purchased. In some embodiments, the computing device 506 mayalternatively or additionally output a haptic effect, e.g., configuredto notify the user 502 that the object 504 was purchased. Thus, video,haptic effects, and/or sound can be used to provide information to theuser 502 about an object 504.

In some embodiments, the computing device 506 may execute a functionand/or output a haptic effect substantially simultaneously with the userinteraction. For example, the computing device 506 may execute multiplefunctions (e.g., determine that a user interaction with the object 504occurred, determine a characteristic of the object 504, and execute afunction based on the characteristic) substantially simultaneously withthe user interaction. The computing device 506 may also output a hapticeffect associated with the function substantially simultaneously withthe user interaction.

FIG. 6 shows another embodiment of a system for object manipulation withhaptic feedback. In this example, the user 602 is wearing a computingdevice 606 comprising a wrist band. The user may gesture on a surface ofthe object 604 to interact with the object. For example, the user mayperform a two finger pinch on, move multiple fingers along, or make acheckmark on a surface of the object 604.

In some embodiments, the computing device 606 detects a gesture and,based on the gesture and/or the object 604, determines a function. Forexample, in the embodiment shown in FIG. 6, the user 602 is interactingwith the object 604 by moving a finger longitudinally across a frontsurface of the object 604. The object 604 may comprise, for example,documents (e.g., medical records). This user may be making this gestureto, for example, print a copy of the documents. Based on this specificgesture, the computing device 606 may output a short vibration. This mayindicate to the user that the computing device 606 is going to print thedocuments if another user interaction is not detected within apredefined time period (e.g., 5 seconds). The computing device 606 mayfurther output a vibration with an increasing magnitude and/or frequencyuntil the time period expires. This may indicate to the user 602 theamount of time left to cancel the print job.

In some embodiments, if the user 602 makes a cancellation gesture beforethe time period expires, the computing device 606 may detect thecancellation gesture and cancel the print job. The cancellation gesturemay comprise a specific gesture, such as a double tap on the surface ofthe object 604, or any gesture at all. The computing device 606 mayoutput a haptic effect (e.g., a jolt) to indicate to the user 602 thatthe print job was cancelled. If the user 602 does not make acancellation gesture within the time period, the computing device 606may cause the documents to be printed. Upon the print job completing,the computing device 606 may output a haptic effect, e.g., configured tonotify the user that the print job is complete.

In some embodiments, the user 602 may use the object 604 as anintermediary for an interaction. The computing device 606 may detectsuch an interaction. For example, the user 602 may tap the object 604(e.g., a document) against a computer monitor or a photocopy machine. Insome embodiments, the computing device 606 may detect the userinteraction and determine a function comprising, for example, copyingtext associated with the object 604 to a virtual clipboard or savingtext associated with the object 604 to a virtual location. The computingdevice 606 may further output a short vibration. This may indicate tothe user that the computing device 606 has successfully copied or savedthe text associated with the object 604.

In some embodiments, a user interaction may comprise causing aninteraction between multiple objects 604. For example, the user 602 maytap the object 604 against another object. For instance, the user 602may tap a stack of papers against a printer. In some embodiments, thecomputing device 606 may detect the interaction between the multipleobjects and determine an associated function. The computing device 606may execute the function. For example, the computing device 606 maydetect the interaction between the stack of papers and the printer andcause data associated with the stack of papers to be printed. In someembodiments, the computing device 606 may also output a haptic effect,e.g., upon the completion of the print job. This may alert the user thatthe print job is complete.

In some embodiments, specific interactions between multiple objects 604can be mapped to specific functions (e.g., via a lookup table). Forexample, an interaction comprising tapping a stack of papers with aminiature model printer can be associated with a function comprisingprinting copies of the papers. In some embodiments, the computing device606 can detect such an interaction and cause copies of the papers to beprinted on a printer. As another example, an interaction comprisingtapping a business card against a phone can be associated with afunction comprising calling a phone number associated with (e.g.,printed on) the business card. In some embodiments, the computing device606 may detect such an interaction and call the phone number.

FIG. 7 shows still another embodiment of a system for objectmanipulation with haptic feedback. In some embodiments, the user 702 mayinteract with an object 704 by making a gesture using the object 704.For example, the user 702 may move the object 704 in real space (e.g.,using the object to draw a letter or number in the air, rotating theobject, tilting the object, moving the object to a specific location,etc.).

In the embodiment shown in FIG. 7, the user 702 is shaking the object704 up and down. The computing device 706 may be configured to executeone or more functions based on the shaking interaction. For example, inone embodiment, the object 704 may comprise an alarm clock. Thecomputing device 706 may detect the shake and set an alarm (e.g.,associated with the time currently output on the alarm clock orpreviously input by the user). The computing device 706 may output ahaptic effect (e.g., a single pulsed vibration) configured to notify theuser that the alarm has been set. If the user 702 thereafter shakes theobject 704 again, the computing device 706 may turn off the alarm. Thecomputing device 706 may further output a haptic effect (e.g., threepulsed vibrations) configured to notify the user that the alarm has beenturned off.

In another embodiment, the object 704 may comprise car keys. Thecomputing device 706 may be remote from the object 704 and configured todetect an interaction with the car keys. For example, the computingdevice 706 may comprise a camera positioned in the user's home andoriented toward the car keys. The user 702 may jiggle the car keys,e.g., as the user 702 is leaving the user's house. The computing device706 may detect the jiggle and transmit text messages or e-mails to apredefined group of friends (e.g., input by the user 702, the user'sfriends, or a third party). The content of the text message or e-mailmay comprise, for example, “I'm on my way!” This may indicate to thefriends that the user is on the user's way to meet the group.

In some embodiments, the user interaction may comprise moving the object704 close to a portion of the user's body (e.g., foot, arm, leg,shoulder, hand, neck, head, back, chest, stomach, thigh, etc.) orcontacting a portion of the user's body with the object 704. Forexample, the user 702 may want to purchase a bottle of soda. The user702 may pick up a bottle of soda and tap his right thigh (e.g., over hiswallet in his right pocket) with the bottle of soda. The computingdevice 706 may detect the tap and determine the quantity of the object704 that the user 702 already has in a refrigerator at home. In someembodiments, the user 702 may have input the quantity into the computingdevice 706. For example, the computing device 706 may execute a shoppinglist application. The user may have input the quantity into the shoppinglist application. The computing device 706 may determine the quantityfrom the shopping list application. In other embodiments, the user 702may have a smart refrigerator or other device capable of determining thecontents of the refrigerator and transmitting the quantity to thecomputing device 706. If the user 702 has a quantity of the productbelow a threshold (e.g., the user 702 has less than three bottles ofsoda), the computing device 706 may not output a haptic effect. If theuser 702 has a quantity of the product above a threshold, the computingdevice 706 may output a buzzing sensation. This may help the user 702make smarter purchasing decisions.

In some embodiments, the user interaction may comprise a throwinggesture. For example, the object 704 may comprise a pen that is low onink. The user 702 may pick up the pen and simulate tossing the pen overthe user's right shoulder. The computing device 706 may detect thethrowing gesture and, based on the user interaction, purchase more pens,e.g., from an online retailer.

In some embodiments, the user interaction may comprise moving the object704 to a specific location. For example, the computing device 706 maydetect the user picking up the object 704 and placing it on or within akiosk (e.g., an information kiosk). In some embodiments, based on theuser 702 putting the object 704 in this specific location, the computingdevice 706 may determine information associated with the object 704,such as whether the object 704 is on sale. If so, the computing device706 may output a haptic effect (e.g., a short vibration).

In some embodiments, the user interaction may comprise moving the object704 through an opening. For example, the computing device 706 may detectthe user picking up the object 704 and exiting the store (e.g., walkingthrough the doorway). In some embodiments, upon the user 702 walkingthrough the store's exit (or through a store's security towers orsensors), the computing device 706 may detect an interaction comprisingthe user taking the object 704. The computing device 706 may execute anassociated function such as, for example, purchasing the object 704 oralerting store security. In some embodiments, the computing device 706may output a haptic effect, e.g., configured to alert the user that theobject 704 has been purchased, that the user 702 is leaving the storewithout purchasing the object 704, or that the store's security guardshave been notified (e.g., a strong vibration).

As another example, the computing device 706 may detect the user pickingup the object 704 and putting the object 704 into the user's shoppingcart or handbag. In some embodiments, based on the user 702 moving theobject 704 through the opening of the user's shopping cart or handbag,the computing device 706 may purchase the object. The computing device706 may also output one or more haptic effects, e.g., associated withthe purchase.

As still another example, the computing device 706 may detect the usermoving the object 704 through a ring-shaped object. In some embodiments,the user may move the object 704 through an object with another shape,such as a square or rectangle. The ring-shaped object may be positioned,for example, at the end of an aisle of shelves in a store. Based on theuser 702 moving the object 704 through the ring's opening, the computingdevice 706 may execute a function. For example, in some embodiments, thecomputing device 706 may determine if the weight of the object 704 isbelow a threshold. If so, the computing device 706 may output a hapticeffect.

In some embodiments, the computing device 706 may detect a plurality ofuser interactions with an object 704 (e.g., making a gesture in front ofthe object 704, contacting the object 704, making a gesture along asurface of the object 704, and making a gesture using the object 704).The computing device 706 may execute one or more functions for eachdetected user interaction. The computing device 706 may further outputone or more haptic effects associated with one or more of the functions.For example, the object 704 may comprise a printer in the user's home.The computing device 706 may detect the user pointing a finger towardthe printer, determine if the amount of ink is low, and output a hapticeffect associated with the amount of ink. The computing device 706 mayalso detect the user tapping three times on the printer, determine anink cartridge type compatible with the printer, and order the inkcartridge.

FIG. 8 shows yet another embodiment of a system for object manipulationwith haptic feedback. In this example, the object comprises anotherperson 804. The user 806 may interact with the person 804 by, forexample, shaking hands with the person 804. A computing device 802associated with the user 806 may detect the interaction and execute afunction based on the interaction.

In some embodiments, the function may comprise determining acharacteristic of the other person 804. The characteristic may comprise,for example, a name, social security number, net worth, height, age,heritage, hair color, nationality, eye color, medical condition, creditscore, gender, credit card number, username (e.g., for a website oraccount), password, temperament, mood, employer, job, hobby, likes,and/or dislikes. The computing device 802 may determine thecharacteristic, for example, by analyzing images of the person 804 froma camera, performing an Internet search (e.g., using the person's 804name), searching a social media website, and/or searching a database(e.g., a public records database). Other examples of methods fordetermining the characteristic are further described below.

In some embodiments, the computing device 802 determines thecharacteristic based on data received (e.g., wirelessly) from anothercomputing device 808 and/or electronic device (e.g., RFID tag). Theother computing device 808 and/or electronic device can be associatedwith the other person 804. For example, the person 804 may be wearing acomputing device 808 comprising a biosensor 810. The biosensor 810 maymeasure a heart rate, temperature, blood pressure, biorhythm, and/orother biological characteristic of the person 804. The computing device808 may transmit this information to the computing device 802 associatedwith the user 806. In some embodiments, the computing device 802 may usethe information as the characteristic. In other embodiments, thecomputing device 802 may use the information to determine acharacteristic about the person 804. For example, the computing device802 may use the information to determine the mood, temperament, oremotional state of the person 804. In one such embodiment, the computingdevice 802 may determine that the person 804 is angry based on, e.g.,information comprising a high blood pressure. In such an embodiment, thecomputing device 808 may output a haptic effect configured to, forexample, warn the user 806 of danger or that the person 804 is angry.

As another example, the computing device 808 may transmit a usernameassociated with a social networking website to the computing device 802.The computing device 802 may consult the social networking website todetermine traits associated with the other person 804.

In other embodiments, the computing device 802 may analyze one or moreimages and/or physical features associated with the person 804 todetermine the characteristic. For example, the computing device 802 maycapture an image of the person 804 using a camera. The computing device802 may use the image to perform facial recognition, read a name tagassociated with the person 804, and/or otherwise identify the person804. The computing device 802 may then communicate with a server (e.g.,for a social network, dating website, search engine, or personalwebsite) to determine additional characteristics about the person 804.

In some embodiments, the computing device 808 executes another functionand/or outputs a haptic effect based on the characteristic. For example,if the other person 804 has the same hobbies as the user 806, thecomputing device 808 may add the other person 804 as a friend on asocial network. As another example, if the other person 804 is the samenationality or religion as the user 806, the computing device 808 mayoutput a haptic effect comprising a vibration.

In some embodiments, the object comprises an animal (e.g., cat, dog,turtle, hamster, ferret, or bird). Upon a user 806 interacting with theanimal, the computing device 802 may determine one or morecharacteristics of the animal (e.g., breed, allergen information,temperament, or mood). The computing device 802 may execute a functionand/or output a haptic effect based on the characteristic. For example,the computing device 802 may read a RFID chip associated with the animaland consult a list of lost animals (e.g., on a website) to determine ifthe animal is lost. If so, the computing device 802 may transmit ane-mail or text message to the owner (or post a message on the website)with the GPS coordinates (or phone number) of the user 806. This mayallow the animal to be reunited with the owner.

Illustrative Methods for Haptically-Enabled Interactions with Objects

FIG. 9 is a flow chart of steps for performing a method for providingobject manipulation with haptic feedback according to one embodiment. Insome embodiments, the steps in FIG. 9 may be implemented in program codethat is executed by a processor, for example, the processor in a generalpurpose computer, a mobile device, or a server. In some embodiments,these steps may be implemented by a group of processors. In someembodiments one or more steps shown in FIG. 9 may be omitted orperformed in a different order. Similarly, in some embodiments,additional steps not shown in FIG. 9 may also be performed. The stepsbelow are described with reference to components described above withregard to computing device 101 shown in FIG. 1.

The method 900 begins at step 902 when the processor 102 receives asensor signal. The processor 102 may receive the sensor signal from theinteraction sensor 132 and/or the additional sensors 130. The sensorsignal may comprise data that is associated with a user interaction withan object. For example, in one embodiment, the data may comprise one ormore images of a store shelf on which an object is sitting. In such anembodiment, the image may comprise features such as the object, theshelf, other nearby objects, and a body part of the user (e.g., a user'sfinger). The user's body part may be contacting or near the object.

The method 900 continues at step 904 when the processor 102 detects auser interaction based on the sensor signal. For example, the computingdevice 101 may be positioned within the object and the interactionsensor 132 may comprise an accelerometer. Upon the user shaking theobject, the processor 102 may detect the shake via sensor signals fromthe accelerometer. As another example, the interaction sensor 132 maycomprise a 3D imaging system oriented toward the object. Upon the usergesturing in front of the object, or along the surface of the object,the processor 102 may analyze images from the 3D imaging system. Uponanalyzing the images, the processor 102 may determine that the pixelscorrelating to the user's hand are oriented in a particular mannerassociated with the user contacting or gesturing near the object.

In some embodiments, the user interaction comprises removing the objectfrom a shelf, table, or other location on which the object ispositioned. For example, the computing device 101 may be embedded in ashelf on which the object sits. An interaction sensor 132 comprising aswitch or a pressure sensor may be positioned on the shelf and below theobject. If the user removes the object from the shelf, the interactionsensor 132 may detect a drop in pressure or a change in the state of theswitch and transmit a sensor signal to the processor 102. Based on thesensor signal, the processor 102 may determine that the user lifted theobject from the shelf. In some embodiments, if the low pressure persistsor the switch remains in a particular state for a predetermined periodof time (e.g., 2 seconds), the processor 102 may determine that the useris holding the object.

In some embodiments, the user interaction comprises positioning thecomputing device 101 within a predefined distance from the object. Forexample, the object may transmit a wireless signal (e.g., an IEEE802.11, Bluetooth, a NFC signal, or RFID signal) to the processor 102(via the network interface device 110). Based on the presence or thestrength of the wireless signal (e.g., if the strength exceeds athreshold), the processor 102 may determine whether the object is withinthe predefined distance (e.g., 3 inches) from the computing device 101.If so, the processor 102 may determine that a user interaction hasoccurred.

As another example, the processor 102 may receive a first set of GPSdata from the interaction sensor 132. The object may also comprise GPSfunctionality and transmit a second set of GPS data to the computingdevice 101. The processor 102 may compare the first set of GPS data tothe second set of GPS data and determine the relative distance betweenthe computing device 101 and the object. If the computing device 101 iswithin a predetermined distance from the object, the processor 102 maydetermine that a user interaction has occurred.

The method 900 continues at step 906 when the processor 102 determines acharacteristic associated with the user interaction. In someembodiments, the processor 102 may analyze signals from the sensor 130and/or the interaction sensor 132 to determine a direction, orientation,pattern, pressure amount, speed, and/or other characteristic of the userinteraction.

For example, in some embodiments, the computing device 101 is positionedwithin the object and the interaction sensor 132 may comprise agyroscope. The processor 102 may determine that the user rotated theobject (and by how much) based on orientation sensor signals from thegyroscope. As another example, in some embodiments, the interactionsensor 132 comprises a 3D imaging system oriented toward the object. Theprocessor 102 may analyze a plurality of images from the 3D imagingsystem to determine the characteristics of a gesture. For instance, theprocessor 102 may analyze a plurality of images and determine that theuser made a “Y” shape in the air with a finger. As still anotherexample, in some embodiments, the computing device 101 is positionedwithin the object and the sensor 130 may comprise an accelerometer. Theprocessor 102 may receive one or more sensor signals from theaccelerometer and determine, for example, that the user is shaking theobject.

The method 900 continues at step 908 when the processor 102 determinesan object associated with the user interaction. In some embodiments, theobject may comprise a product in a store (e.g., a box of cereal), a tool(e.g., a hammer or screw driver), a medical device (e.g., a trocar,needle, or heartbeat monitor), an automobile, a human, an animal, etc.

In some embodiments, the processor 102 may associate a user interactionwith a particular object based on the proximity of the computing device101 to the object. Examples include using GPS data and the strength of awireless signal (e.g., as described in step 902 above) to determinewhether the computing device 101 is within a predefined distance to theobject. If so, the processor 102 may associate the object with the userinteraction.

In some embodiments, the processor 102 associates a user interactionwith a particular object based on the proximity of the user interactionto the object. For example, a user may gesture with a finger in front ofan object. The interaction sensor 132 may detect the gesture andtransmit a sensor signal to the processor 102. The processor 102 mayanalyze the sensor signal to determine the distance between the fingerand the object. If the finger is within a particular distance (e.g., 6inches) from the object, the processor 102 may associate the gesturewith the object.

In some embodiments, the processor 102 associates a user interactionwith a particular object based on a direction or orientation of the userinteraction. For example, a user may point at an object the user isinterested in purchasing. The interaction sensor 132 may detect thegesture and transmit a sensor signal to the processor 102. The processor102 may analyze the sensor signal and determine that the point gestureis directed toward the object. Based on the direction of the gesture,the processor 102 may associate the object with the point interaction.

In some embodiments, the processor 102 associates a user interactionwith a particular object by comparing sensor data from the object tosensor data from the computing device 101. For example, a user may weara wristband comprising the computing device 101. The user may pick upand shake an object. The processor 102 may receive a first set ofaccelerometer data associated with the shake from an interaction sensor132 within the wristband. The processor 102 may also receive a secondset of accelerometer data from an accelerometer within the packaging ofthe object or the object itself (e.g., the object may wirelesslycommunicate the accelerometer data to the computing device 101). Theprocessor 102 may compare the two sets of accelerometer data. Forexample, the processor 102 may compare the time stamps of peaks in thetwo sets of accelerometer data. If the time difference between the twotime stamps is below a threshold (e.g., 0.5 s), the processor 102 maydetermine that the shake was with respect to that particular object.

As another example, a user may pick up an object off a shelf. The shelfmay comprise a computing device 101 configured detect the object'sremoval and transmit (e.g., wirelessly) a time stamp associated withwhen the object was removed from the shelf. The processor 102 mayreceive the time stamp and compare the time stamp with a time stampassociated with data (e.g., an image) from interaction sensor 132. Ifthe time difference between the two time stamps is below a threshold,the processor 102 may determine that the user picked up that particularobject.

The method 900 continues at step 910 when the processor 102 determines acharacteristic associated with the object. In some embodiments, theprocessor 102 may rely on programming in the object interactiondetection module 124 to determine the characteristic. In someembodiments, the processor 102 determines a characteristic associatedwith the object based on the characteristic associated with the userinteraction. For example, the processor 102 may determine differentcharacteristics associated with an object in response to different userinteractions. For example, the processor 102 may determine themanufacturer on the label of a food product when the user the user tapsthe food product. The processor 102 may determine the color of a foodproduct when the user moves a finger along the surface of the foodproduct in a “C” shape.

In some embodiments, processor 102 determines a characteristicassociated with the object based on a wired or wireless signal (e.g.,from another computing device 101). For example, the object may comprisea computing device 101 or the shelf may comprise a computing device 101.The object or the shelf may transmit information about the object whichcan be received by the processor 102. The information may include, forexample, an identifier (e.g., a name, QR code, bar code, QR code, RFIDcode, or unique identifier), classification (e.g., the type of product,manufacturer, producer, or brand), price, discount, coupon, or function(e.g., what the product does or may be used for) associated with theobject. For example, if the object is a shampoo bottle, the informationmay comprise exactly which bottle of shampoo it is (e.g., number 3 onthe shelf), what kind of product it is (e.g., shampoo), what specificproduct it is (e.g., Hair Company's shampoo for fine hair, or abarcode), or what brand of product it is (e.g., Hair Company brand).

In some embodiments, the processor 102 determines the characteristicusing the Internet or a LAN. The processor 102 may communicate with oneor more servers, databases, and/or webpages via the Internet or a LAN todetermine the characteristic. For example, the processor 102 querymultiple databases over the Internet (e.g., using the object's bar code)to determine the number of stores proximate to the user (e.g., within 10miles) that sell the object. The processor 102 may also determine whichstore has the cheapest price.

The method 900 continues at step 912 when the processor 102 determines afunction. In some embodiments, the processor 102 may rely on programmingin the function determination module 125 to determine thecharacteristic.

In some embodiments, the processor 102 determines a function based onthe user interaction. For example, the processor 102 may determinedifferent functions in response to different user interactions. Forinstance, upon the user tapping on a poster of a band, the processor 102may determine a function comprising downloading a MP3 associated withthe band. The processor 102 may determine a function comprisingretrieving the lyrics to a popular song when the user swipes a fingeralong the surface of the music poster.

In some embodiments, processor 102 determines a function based on acharacteristic of the object. For example, the processor 102 maydetermine a function based on the type of the object. For instance, inone embodiment, upon a user tapping on a book, the processor 102 maydetermine a function comprising adding the book to a reading list. Inanother embodiment, upon the user tapping on a battery, the processor102 may determine a function comprising ordering a new battery.

In some embodiments, the computing device 101 may store associated“function profiles” in which a user can determine and save in memory 104a “profile” of the functions the user would like associated withparticular objects and/or user interactions. For example, in oneembodiment, a user can select from a list of options which function theuser would like associated with the tapping on a document, swiping afinger along a surface of the document, and rotating the document. Insome embodiments, the list may comprise, for example, downloading anelectronic copy of the document, printing the document, and e-mailing anelectronic copy of the document. In such an embodiment, the processor102 may consult with the user's function profile to determine afunction. For example, if the user's function profile associates tappingon the document with downloading an electronic copy of the document, inresponse to the user tapping on the document, the processor 102 maydetermine a function comprising downloading the electronic copy of thedocument.

In some embodiments, the processor 102 determines the function using theInternet or a LAN. The processor 102 may communicate with one or moreservers, databases, other computing devices (e.g., within the object,the object's packaging, a shelf, or a store), and/or webpages via theInternet or a LAN to determine the function. For example, the processor102 may query a server over the Internet (e.g., using the object's barcode) to determine which functions are available and/or associated witha particular object. The server may transmit a list of availablefunctions for the particular object to the computing device 101. Thecomputing device 101 may determine a function based on the list ofavailable functions.

As another example, the computing device 101 may receive data (e.g.,from a server) that maps user interactions to functions for a particularobject. For example, the computing device 101 may receive data thatmaps, for a particular object, a swipe gesture to a function comprisingpurchasing the object. Upon the computing device 101 detecting a userinteraction, the computing device 101 may determine the correspondingfunction based on the data. This may allow multiple users interactingwith the particular object to have a consistent experience, because eachcomputing device 101 will execute similar functions based on similaruser interactions.

As still another example, the processor 102 may determine a functionbased on a characteristic of the object received via a wired or wirelesssignal (e.g., from another computing device 101). For example, a shelfcomputing device may transmit information about the object, which can bereceived by the processor 102. The information may include, for example,an identifier (e.g., a name, QR code, bar code, QR code, RFID code, orunique identifier), classification (e.g., the type of product,manufacturer, producer, or brand), price, discount, coupon, or function(e.g., what the product does or may be used for) associated with theobject. For example, if the object is a shampoo bottle, the informationmay comprise exactly which bottle of shampoo it is (e.g., number 3 onthe shelf), what kind of product it is (e.g., shampoo), what specificproduct it is (e.g., Hair Company's shampoo for fine hair, or abarcode), or what brand of product it is (e.g., Hair Company brand).Based on the characteristic, the processor 102 may determine one or morefunctions.

The method 900 continues at step 914 when the processor 102 executes thefunction. In some embodiments, the processor 102 may execute thefunction by executing one or more sub-functions. For example, if thefunction comprises purchasing an object via the Internet for thecheapest available price, the processor 102 may communicate withmultiple servers (e.g., a search engine and one or more websites) todetermine all of the stores through which the object can be purchasedonline. The processor 102 may also determine which of the stores has thecheapest price. The processor 102 may further purchase the object fromthe store with the cheapest price (e.g., via the store's website oranother server).

The method 900 continues at step 916 when the processor 102 determines ahaptic effect. In some embodiments, the processor 102 may determine ahaptic effect associated with the function. For example, the processor102 may determine a haptic effect associated with a type of thefunction. For instance, in one embodiment, the processor 102 maydetermine a haptic effect comprising a short vibration if the functioncomprises adding an entry to a list. In such an embodiment, theprocessor 102 may determine a haptic effect comprising three shortvibrations if the function comprises removing an entry from a list. Thismay indicate to the user which function was performed (or will beperformed in the future).

In some embodiments, the processor 102 determines a haptic effectconfigured to indicate that the function has been executed. For example,the processor 102 may determine a haptic effect comprising a strokingsensation once the function has executed. This may indicate to the userthat the function was performed. In some embodiments, the processor 102may determine different haptic effects based on the result of executingthe function. For example, the processor 102 may output a differenthaptic effect if the function was successfully executed than if thefunction was not successfully executed.

In some embodiments, the processor 102 determines a haptic effectconfigured to indicate whether an object can be manipulated to inputdata and/or execute a specific function. For instance, the processor 102may detect a user picking up an object and output a haptic effect (e.g.,a vibration) if, for example, the object can be purchased by performinga gesture with the object (e.g., shaking the object). The processor 102may not output a haptic effect if the object cannot be purchased byperforming a gesture with the object. Thus, in some embodiments, thehaptic effect may indicate to the user whether the object can bemanipulated to perform a specific task.

In some embodiments, the processor 102 determines a haptic effectassociated with the progress of a function being executed. For example,the processor 102 may determine a haptic effect comprising a pulsedvibration with a frequency and/or period associated with the progress ofa download or the playback of a media file. This may indicate to theuser how much time is left until the function is done executing or themedia file is done playing. In some embodiments, the processor 102determines a haptic effect associated with the cancellation of afunction (e.g., prior to being executed or while the function is beingexecuted). For example, the processor 102 may determine a haptic effectcomprising a stinging sensation if the user cancels the execution of afunction prior to (or during) the execution of the function.

In some embodiments, the processor 102 determines a haptic effectconfigured to indicate that the computing device 101 is awaitingadditional user input and/or a status of the function. For example, thecomputing device 101 may detect an interaction with an object. Based onthe interaction, the computing device 101 may determine a functioncomprising purchasing the object. The computing device 101 may enter allthe user's details into a purchase form and wait for the user to provideconfirmation to finalize the purchase. The computing device 101 mayoutput a haptic effect (e.g., a vibration) configured to alert the userthat the computing device 101 is waiting for the user's confirmation.Upon the user performing another interaction with the object (e.g.,rotating the object 90 degrees), the computing device 101 may finalizethe purchase. Once the purchase is complete, the computing device 101may output another haptic effect, e.g., to indicate that the purchase iscomplete.

In some embodiments, the processor 102 determines a haptic effect basedon a characteristic associated with the user interaction. For example,the processor 102 may determine the haptic effect based the type,location, duration, or other characteristics of the user interaction.For instance, the processor 102 may determine a haptic effect comprisinga vibration if the user contacts a specific portion of a product, suchas the nutritional label. As another example, the processor 102 maydetermine a haptic effect if the user interaction comprises a swipealong a surface of the object, and no haptic effect if the userinteraction comprises tapping on the object.

In some embodiments, the computing device 101 may store associated“haptic profiles” in which a user can determine and save in memory 104 a“profile” of the haptic effects the user would like associated withparticular functions and/or user interactions. For example, in oneembodiment, a user can select from a list of options which haptic effectthe user would like associated with functions such as determining theprice of an object, printing a document associated with an object,opening a website associated with an object, or determining a materialof an object. In some embodiments, the list may comprise, for example,haptic effects such as low magnitude vibration, pulsed vibration,high-magnitude vibration, or a simulated texture. In some embodiments,the processor 102 may consult with the user's haptic profile todetermine which haptic effect to generate. For example, if the user'shaptic profile associates locally produced products with a haptic effectcomprising a low-frequency vibration, in response to the user contactinga locally made chocolate bar, the processor 102 may determine a hapticeffect comprising a low-frequency vibration.

In some embodiments, the processor 102 may determine a default hapticeffect. For example, if the processor 102 is unable to perform afunction (e.g., unable to execute an application, purchase a product, ordownload a file), the processor 102 may determine a haptic effectcomprising a vibration. This may alert the user that the operation couldnot be performed.

For example, the computing device 101 may be configured to output a joltif the user interacts with a product that is too expensive (e.g., theprice is above a threshold). The computing device 101 may be configuredto output no haptic effect if the user interacts with a product that isnot too expensive. In such an embodiment, if the processor 102 cannotdetermine the price of the product (e.g., because it cannot connect tothe Internet), the processor 102 may determine a haptic effectcomprising three pulsed vibrations. Thus, rather than the computingdevice 101 outputting no haptic effect, which could falsely lead theuser to believe the product is not too expensive, the three pulsedvibrations can alert the user that the processor 102 was unable todetermine the price. The user can then read the price tag or otherwisedetermine whether the product is too expensive.

In some embodiments, the processor 102 determines a haptic effect basedon a characteristic of the user. For example, if the processor 102determines that the user is a male, the processor 102 may determine adifferent haptic effect than if the processor 102 determines that theuser is a female. As another example, the processor 102 may determine ahaptic effect if the user is under 40 years old than if the user is over40 years old. In this manner, haptic effects can be customized fordifferent demographics, which may improve a user's response to thehaptic effect.

In some embodiments, the processor 102 determines a plurality of hapticeffects. Each of the plurality of haptic effects may be associated witha different function. For example, the processor 102 may determine ahaptic effect associated with a determining whether the size of theobject is below a threshold (e.g., a long vibration). The processor 102may also determine a haptic effect associated with adding the object toa shopping list. Based on the plurality of haptic effects, the user maybe able to determine that multiple different functions have beenexecuted (e.g., that the size of the object was determined and that theobject has been added to a shopping list), for example, without havingto measure the object or interact with a touchscreen user interface(which may be tedious and time consuming).

In some embodiments, the processor 102 determines that a specific hapticeffect has a higher priority than another haptic effect, and thus tooutput only the high priority effect. For example, in the above size andshopping list example, the processor 102 may determine that the hapticeffect associated with the size has a higher priority than other effects(e.g., the haptic effect associated with adding the object to a shoppinglist), and thus output only the size haptic effect. Alternatively, theprocessor 102 may determine that only the most intense effect should beoutput. Thus, in some embodiments, the processor may determine a lowintensity vibration and a high intensity vibration, but output only thehigh intensity vibration.

In some embodiments, the processor 102 may determine a haptic effectconfigured to make it easier or harder to interact with the object. Forexample, the object may be positioned on a shelf. Upon a userinteracting with the object, the processor 102 may determine that theobject comprises a drug harmful to the user. Based on the harmful drug,the processor 102 may transmit a signal to a packaging computing device.The packaging computing device may receive the signal and output ahaptic effect configured to lower the perceived coefficient of frictionon the surface of the object's packaging. This may make it challengingfor the user to grip or grasp the object to pick it up. Further, in someembodiments, the object (e.g., packaging of the object) may have a metalbottom. Based on the harmful drug, the processor 102 may transmit asignal to a shelf computing device. The shelf computing device mayreceive the signal and actuate an electromagnetic device configured toapply a strong magnetic field to the metal bottom of the object. Thismay make it challenging for the user to lift the object off the shelf(e.g., to purchase it).

As another example, the object's packaging may be deformed aroundanother object, such as a shelf (e.g., to secure the object to theshelf). For instance, the shelf may comprise a bar around which thepackaging (or object) is bent to secure the object to the bar. In someembodiments, upon the user interacting with the object, the processor102 may transmit data (e.g., a purchase receipt code, name, credit cardnumber, or customer identifier) to a packaging computing devicepositioned in the object's packaging. The data may indicate that theuser already purchased the item (e.g. online), or may otherwise beassociated with a purchase of the product (e.g., the data may include acredit card number which the object may use to conduct the purchase inreal time). Based on the data, in some embodiments, the packagingcomputing device may output a haptic effect configured to deform theshape of the packaging. For example, the packaging computing device mayoutput a haptic effect configured to unbend the packaging from aroundthe bar (e.g., to de-secure the object from the shelf). This may allowthe user to take the object.

The method 900 continues at step 918 when the processor 102 outputs thehaptic effect. The processor 102 may transmit a haptic signal associatedwith the haptic effect to haptic output device 118, which outputs thehaptic effect. The haptic effect may comprise a texture (e.g., sandy,bumpy, or smooth), a vibration, a change in a perceived coefficient offriction, a change in temperature, a stroking sensation, anelectro-tactile effect, or a deformation (i.e., a deformation of asurface associated with the computing device 101).

Additionally or alternatively, in some embodiments, the computing device101 may output a sound and/or information on a display. The sound and/orinformation may be associated with a characteristic of the object and/ora function. For example, the user may tap on a toy. The computing device101 may determine the name of the toy and, based on the name, whether toexecute a function comprising outputting the name of the toy on thedisplay. The computing device 101 may also determine if the toy is on abirthday list (e.g., made by the user's friends). If so, the computingdevice 101 may execute a function comprising outputting a sound. Thesound may comprise a beep. This may notify the user that the user hasinteracted with an item on the birthday list.

Advantages of Haptically-Enabled Interactions with Objects

There are numerous advantages to haptically-enabled interactions withobjects. Such systems may provide information about an object quicklyand efficiently. For example, a user can tap on an object andsubstantially simultaneously receive information about the price of anobject (e.g., via haptic effects). This may be quicker and easier than,for example, looking up such information using a mobile device orconsulting with a store employee.

In some embodiments, haptically-enabled interactions with objects canallow a user to execute one or more functions (e.g., digital operations)quickly and easily. For example, a computing device may detect a userinteraction with a laptop computer (e.g., a user drawing an “A” with afinger along the surface of the laptop computer's monitor). Based on theuser interaction, the computing device may transmit one or more signalsto the laptop computer configured to cause the laptop computer toperform a hardware analysis. The computing device may receive data fromthe laptop computer associated with any failed hardware componentsdetermined from the hardware analysis. The computing device may furtherorder one or more of the failed hardware components via the Internet.Thus, the computing device can quickly and easily perform a plurality ofdigital operations that may be cumbersome for a user.

In some embodiments, haptically-enabled interactions with objects canallow a user to determine a status of one or more functions and/or thecomputing device. For example, a user may want to download a copy of asong playing on the radio. While the song is playing, the user may tapon a MP3 player twice. The computing device may detect the two taps,begin recording audio via a microphone, and analyze the audio toidentify the song. If the computing device cannot identify the song, thecomputing device may output a buzz sensation, e.g., to indicate to theuser that the computing device cannot download the song. If thecomputing device can identify the song, the computing device may outputa high-frequency vibration, e.g., to indicate to the user that thecomputing device has successfully identified the song. Thereafter, insome embodiments, the computing device may wait to receive anothergesture to confirm the download. The computing device may output threeshort vibrations indicating that the computing device is waiting foradditional user input. The computing device may detect a third tap bythe user and download the MP3 from a server. Upon the downloadfinishing, the computing device may output a jolt sensation, e.g., toindicate to the user that the download is complete.

General Considerations

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail to avoid obscuringthe configurations. This description provides example configurationsonly, and does not limit the scope, applicability, or configurations ofthe claims. Rather, the preceding description of the configurations willprovide those skilled in the art with an enabling description forimplementing described techniques. Various changes may be made in thefunction and arrangement of elements without departing from the spiritor scope of the disclosure.

Also, configurations may be described as a process that is depicted as aflow diagram or block diagram. Although each may describe the operationsas a sequential process, many of the operations can be performed inparallel or concurrently. In addition, the order of the operations maybe rearranged. A process may have additional steps not included in thefigure. Furthermore, examples of the methods may be implemented byhardware, software, firmware, middleware, microcode, hardwaredescription languages, or any combination thereof. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, in which other rules may takeprecedence over or otherwise modify the application of the invention.Also, a number of steps may be undertaken before, during, or after theabove elements are considered. Accordingly, the above description doesnot bound the scope of the claims.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

Embodiments in accordance with aspects of the present subject matter canbe implemented in digital electronic circuitry, in computer hardware,firmware, software, or in combinations of the preceding. In oneembodiment, a computer may comprise a processor or processors. Theprocessor comprises or has access to a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs including a sensor samplingroutine, selection routines, and other routines to perform the methodsdescribed above.

Such processors may comprise a microprocessor, a digital signalprocessor (DSP), an application-specific integrated circuit (ASIC),field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example tangible computer-readable media, that may storeinstructions that, when executed by the processor, can cause theprocessor to perform the steps described herein as carried out, orassisted, by a processor. Embodiments of computer-readable media maycomprise, but are not limited to, all electronic, optical, magnetic, orother storage devices capable of providing a processor, such as theprocessor in a web server, with computer-readable instructions. Otherexamples of media comprise, but are not limited to, a floppy disk,CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configuredprocessor, all optical media, all magnetic tape or other magnetic media,or any other medium from which a computer processor can read. Also,various other devices may comprise computer-readable media, such as arouter, private or public network, or other transmission device. Theprocessor, and the processing, described may be in one or morestructures, and may be dispersed through one or more structures. Theprocessor may comprise code for carrying out one or more of the methods(or parts of methods) described herein.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude inclusion of suchmodifications, variations and/or additions to the present subject matteras would be readily apparent to one of ordinary skill in the art.

1-20. (canceled)
 21. A computing device comprising: a processor; and amemory device comprising instructions executable by the processor tocause the processor to: detect an interaction associated with a physicalobject that is separate from the computing device using one or moresensors, wherein the interaction includes a gesture or a contact;determine a characteristic of the physical object by communicating witha remote device that is different from the physical object; determine ahaptic effect based on the characteristic of the physical object; andtransmit a haptic signal configured to cause a haptic output device tooutput the haptic effect.
 22. The computing device of claim 21, whereinthe interaction is a user interaction with the physical object itself.23. The computing device of claim 22, wherein user interaction includesa user contacting the physical object.
 24. The computing device of claim21, wherein the physical object is a product for sale, and wherein theinteraction is a user interaction with packaging for the product. 25.The computing device of claim 24, wherein the user interaction includesa user contacting the packaging.
 26. The computing device of claim 21,wherein the interaction includes a user gesture, the user gesture beinga predetermined movement oriented toward the physical object and distantfrom the physical object.
 27. The computing device of claim 21, whereinthe interaction involves moving the physical object.
 28. The computingdevice of claim 27, wherein the interaction involves positioning thephysical object in a carrying device.
 29. The computing device of claim21, wherein the computing device is part of a wearable device.
 30. Thecomputing device of claim 29, wherein the wearable device is a ring,watch, or wristband.
 31. A method comprising: detecting, by a processorof a computing device, an interaction associated with a physical objectthat is separate from the computing device using one or more sensors,wherein the interaction includes a gesture or a contact; determining, bythe processor, a characteristic of the physical object by communicatingwith a remote device that is different from the physical object;determining, by the processor, a haptic effect based on thecharacteristic of the physical object; and transmitting, by theprocessor, a haptic signal configured to cause a haptic output device tooutput the haptic effect.
 32. The method of claim 31, wherein theinteraction is a user interaction with the physical object itself. 33.The method of claim 32, wherein user interaction includes a usercontacting the physical object.
 34. The method of claim 31, wherein thephysical object is a product for sale, and wherein the interaction is auser interaction with packaging for the product.
 35. The method of claim34, wherein the user interaction includes a user contacting thepackaging.
 36. The method of claim 31, wherein the interaction includesa user gesture, the user gesture being a predetermined movement orientedtoward the physical object and distant from the physical object.
 37. Themethod of claim 31, wherein the interaction involves moving the physicalobject.
 38. The method of claim 37, wherein the interaction involvespositioning the physical object on or inside a carrying device.
 39. Themethod of claim 31, wherein the computing device is part of a wearabledevice.
 40. The method of claim 39, wherein the wearable device is aring, watch, or wristband.
 41. A non-transitory computer-readable mediumcomprising program code that is executable by a processor of a computingdevice to cause the processor to: detect an interaction associated witha physical object that is separate from the computing device using oneor more sensors, wherein the interaction includes a gesture or acontact; determine a characteristic of the physical object bycommunicating with a remote device that is different from the physicalobject; determine a haptic effect based on the characteristic of thephysical object; and transmit a haptic signal configured to cause ahaptic output device to output the haptic effect.